This plugin for Inspect allows you to use virtual machines, running within one or more Proxmox instances, as sandboxes.
Add this using uv,
uv add git+ssh://git@github.com/UKGovernmentBEIS/inspect_proxmox_sandbox.git
or with Poetry,
poetry add git+ssh://git@github.com/UKGovernmentBEIS/inspect_proxmox_sandbox.git
This plugin assumes you already have one or more Proxmox instances set up, and that you have admin access to them.
Proxmox 9 or later is required; 9.2 or later is recommended for improvements in read_file.
Your Proxmox instance(s) must allow additional storage types in local from the default.
You can run this on your Proxmox node to configure them:
pvesh set /storage/local -content iso,vztmpl,backup,snippets,images,rootdir,importSDN requires you to configure dnsmasq, see the Proxmox SDN documentation. Note, the commands on that page must be run on the Proxmox node, not your local machine.
If you don't already have a Proxmox instance, see CONTRIBUTING.md for supported setup paths (local Ubuntu 24.04 host, or EC2 with nested virtualization).
By default a sandbox VM can reach the Proxmox host's own services — the API
(pveproxy, port 8006), SSH, etc. — via the SDN gateway, the vmbr0 IP, or the
host's external NIC. Forwarded traffic can also reach cloud instance metadata
services. For cyber evals especially, you want those blocked so agents can't
attack the Proxmox or cloud control planes.
This is configured on the host at provisioning time, not by this library — it
needs the host's live routing table to know which interface external API/SSH
traffic arrives on, which only the host itself can tell you reliably. The
provisioning scripts in this repo (scripts/virtualized_proxmox/build_proxmox_auto.sh
and scripts/ec2/userdata.sh) set it up automatically, so hosts you create with
them are isolated out of the box.
If you provision Proxmox some other way, configure equivalent persistent rules
on the node. The Proxmox rules accept management ports only on the
default-route interface (where external callers arrive) and leave SDN DNS/DHCP
open. The remaining rules enforce RFC 3927 section 7: a router must not forward IPv4 link-local (169.254.0.0/16) traffic.
Dropping it stops a sandbox guest reaching the host's
cloud metadata service — and any other link-local endpoint. The
destination drop goes in raw PREROUTING (host requests are OUTPUT, never
PREROUTING, so the host keeps its own access); the source drop goes in
FORWARD, not PREROUTING, so the host's own replies (e.g. an IMDS or
169.254.169.253 DNS response, delivered to INPUT) are left intact — a
raw PREROUTING -s rule would drop them and break the host.
We also recommend disabling forwarding of IPv6 for VMs, unless you really know what you are doing.
NIC=$(ip route show default | awk '{print $5}' | head -1)
pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT --proto tcp --dport 8006 --iface "$NIC" --enable 1
pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT --proto tcp --dport 22 --iface "$NIC" --enable 1
pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT --proto udp --dport 53 --enable 1
pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT --proto tcp --dport 53 --enable 1
pvesh create /nodes/$(hostname)/firewall/rules --type in --action ACCEPT --proto udp --dport 67 --enable 1
pvesh set /nodes/$(hostname)/firewall/options --enable 1
pvesh set /cluster/firewall/options --enable 1
iptables -w -t raw -C PREROUTING -d 169.254.0.0/16 -j DROP 2>/dev/null \
|| iptables -w -t raw -I PREROUTING 1 -d 169.254.0.0/16 -j DROP
iptables -w -C FORWARD -s 169.254.0.0/16 -j DROP 2>/dev/null \
|| iptables -w -I FORWARD 1 -s 169.254.0.0/16 -j DROP
sysctl -w net.ipv6.conf.default.disable_ipv6=1 # new SDN bridges come up v6-off
if command -v ip6tables >/dev/null; then
ip6tables -w -C FORWARD -j DROP 2>/dev/null || ip6tables -w -A FORWARD -j DROP
fiMost clouds (AWS, GCP, Azure, Oracle, DigitalOcean) serve metadata from
169.254.169.254, covered above. Two providers sit outside the link-local
range: Alibaba Cloud uses 100.100.100.200, and Azure exposes the
WireServer at 168.63.129.16 (guest-agent goal state / extension settings). On
those clouds, add a -d <ip>/32 -j DROP raw-table rule for each — the host
keeps access since its own traffic doesn't traverse PREROUTING.
You must persist these rules across reboots (the bundled provisioning scripts do this, if you need an example.)
These work under either firewall backend (pve-firewall or the nftables
proxmox-firewall); the latter won't touch these chains. On iptables-legacy
hosts they won't show in nft list ruleset — use iptables -t raw -S / -S FORWARD.
The provisioning scripts also install but don't activate an egress lockdown
for sandbox guests. When active, all traffic forwarded between guests and
every interface carrying a default route is dropped, and the per-zone SDN
dnsmasq instances are stopped from recursing to any upstream resolver.
Together these close both direct egress and the DNS-resolution channel a guest
could otherwise tunnel through (names no longer resolve beyond the internal
vnets). Unaffected: guest↔guest traffic across vnets (it never crosses the
management NIC) and the host's own egress and DNS (package installs, cloud
agents, SSH).
Guests must not need egress to boot: the built-in VM template bake (first use
of a built_in image on a host) installs packages from inside the guest, so
it must happen before the lockdown is applied. Sandbox VMs cloned from an
already-baked template boot fine.
The lockdown is gated on a marker file that provisioning doesn't create, so hosts are unrestricted by default. To restrict a running host:
touch /etc/inspect-proxmox-egress-lockdown
systemctl start inspect-proxmox-egress-lockdown.serviceTo open it up again:
rm /etc/inspect-proxmox-egress-lockdown
systemctl start inspect-proxmox-egress-lockdown.serviceA systemd timer re-runs the lockdown every minute. Each run re-inserts the drop rules at the top of their chains and garbage-collects rules from earlier runs, so rules that another process removed or reordered (e.g. a firewall reload) are repaired within a minute, and removing the marker converges to a fully clean state without waiting for a reboot.
The lockdown fails closed. If no default-route interface can be found, the
script applies a blanket drop on all forwarded traffic — which also cuts
guest↔guest traffic — and fails the unit. Any unit failure triggers a
fail-deadly halt: pveproxy and pvedaemon are stopped and runtime-masked,
taking the Proxmox API (and this library's ability to run samples on the host)
down rather than risking guests with open egress. SSH is unaffected, so an
operator can investigate with
journalctl -u inspect-proxmox-egress-lockdown.service and recover with
systemctl unmask --runtime pveproxy.service pvedaemon.service && systemctl start pveproxy.service pvedaemon.service; a reboot also clears the runtime
mask.
The drop rules live in the mangle table's FORWARD chain, which is evaluated
before every filter-table rule, so activating the lockdown also cuts off guest
connections that are already established (e.g. a download started beforehand).
Neither firewall backend touches the mangle table, so there are no coexistence
conflicts.
The DNS side works by blanking the upstream resolver file (/run/dnsmasq/resolv.conf)
the SDN dnsmasq instances forward through and reloading them, so they keep
answering internal names but refuse everything else. The previous upstream is
backed up and restored when the marker is removed, so opening the host back up
also restores guest DNS. As a firewall backstop, host-originated dnsmasq
traffic out of the default-route interfaces is also dropped, so upstream
recursion stays blocked even if something rewrites the resolver file (it lives
on tmpfs and is recreated at boot) before the timer re-blanks it.
The lockdown is only a strong control for the SDN topology this library generates: it covers traffic forwarded between guest vnets and the host's default-route interfaces. Pre-existing bridges wired straight to other NICs, template VMs with extra network devices left attached, interfaces without a default route, and tunnels originating on the host itself are out of scope.
Set the following environment variables (e.g. in a .env file):
PROXMOX_HOST=[IP address or domain name of the host]
PROXMOX_PORT=[port, e.g 8006]
PROXMOX_USER=[user, usually 'root']
PROXMOX_REALM=[authentication realm, usually 'pam' unless you have configured custom auth]
PROXMOX_PASSWORD=[password]
PROXMOX_NODE=[node name, usually 'proxmox']
PROXMOX_VERIFY_TLS=[1 = verify, 0 = do not verify]
PROXMOX_IMAGE_STORAGE=[storage pool for VM disk images, usually 'local-lvm']
To run evals across multiple Proxmox servers, create a JSON config file and point to it with PROXMOX_CONFIG_FILE:
export PROXMOX_CONFIG_FILE=/path/to/instances.jsoninstances.json:
{
"instances": [
{
"instance_id": "proxmox-1",
"pool_id": "ubuntu-ami-123",
"host": "10.0.1.10",
"port": 8006,
"user": "root",
"user_realm": "pam",
"password": "secret",
"node": "pve1",
"verify_tls": false
},
{
"instance_id": "proxmox-2",
"pool_id": "ubuntu-ami-123",
"host": "10.0.1.11",
"port": 8006,
"user": "root",
"user_realm": "pam",
"password": "secret",
"node": "pve2",
"verify_tls": false
}
]
}Instances with the same pool_id form a pool. Each eval sample acquires one instance from its pool, uses it exclusively, and releases it back when done. Concurrency is automatically limited to the total number of instances.
extra_headers on an instance adds HTTP headers to every request sent to that instance's Proxmox API, including file uploads. See schema.py for details.
Header values are treated as secrets and redacted from logs. The Proxmox authentication headers (Cookie, CSRFPreventionToken) cannot be overridden.
Here is a full example sandbox configuration.
Note that some of the fields (e.g. subnets) are tuples, so the trailing comma is vital if there is only a single item in the tuple.
Most tools use only the first sandbox, so you should list the one you want the agent to operate from first.
Virtual machines must have the qemu-guest-agent installed, unless they are not sandboxes. At least one VM in the configuration must be a sandbox.
sandbox=SandboxEnvironmentSpec(
"proxmox",
ProxmoxSandboxEnvironmentConfig(
# Storage pool for VM disk images. Defaults to PROXMOX_IMAGE_STORAGE env var
# or "local-lvm" if not set.
image_storage="local-lvm",
# When using PROXMOX_CONFIG_FILE with multiple instances, set this to select
# which pool to use (must match a pool_id in the config file).
# Not needed for single-instance setups.
# instance_pool_id="ubuntu-ami-123",
vms_config=(
VmConfig(
# A virtual machine that this provider will install and configure automatically.
vm_source_config=VmSourceConfig(
built_in="ubuntu24.04" # currently supported: "ubuntu24.04", "debian13", "kali2025.4"; see schema.py
),
name="romeo", # name is optional, but recommended - it will be shown in the Proxmox GUI and registered as the Inspect sandbox environment identifier. Must be a valid DNS name.
ram_mb=512, # optional, default is 2048 MB
vcpus=4, # optional, default is 2. No attempt is made to check that this will fit in the Proxmox host.
uefi_boot=True, # optional, default is False. Generally only needed for Windows VMs.
is_sandbox=False, # optional, default is True. A virtual machine that is not a sandbox; the qemu-guest-agent need not be installed.
disk_controller="scsi", # optional, default will be SCSI. Can also use "ide" for older VM images.
nic_controller="virtio", # optional, default will be VirtIO. Can also use "e1000" for older VM images.
cpu="host", # optional, default "host". The qemu CPU model (e.g. "host", "qemu64", "x86-64-v2"). Older guest kernels (notably FreeBSD/pfSense) can panic on nested virtualization with "host"; use "qemu64" for those.
firewall=True, # optional, default is False. Enables the Proxmox firewall on all NICs for VM isolation.
# If you have more than one VNet, assign the VM to the VNet via nics.
# You can assign more than one, to give the VM more than one network interface.
# If you leave this blank, your VM will be assigned to the first VNet.
nics=(
VmNicConfig(
# This alias *must* match the alias in one of the VnetConfigs
vnet_alias="my special vnet",
# Specifying a MAC address is optional - only needed if you
# are doing fancy things with DHCP in your eval, or if you
# want to assign a static IP address
mac="00:16:3d:1d:eb:a0",
# Specifying a static IPv4 address is optional. If provided,
# a DHCP static mapping (host reservation) will be created.
# Note: requires a MAC address to be specified as well.
# Please read the notes in README.md for Proxmox server patching requirements
ipv4=ip_address("192.168.20.10")
),
)
# extra_proxmox_native_config = dict() TODO
),
# A virtual machine from a local OVA, which will be uploaded from here to the Proxmox server.
VmConfig(
vm_source_config=VmSourceConfig(
ova=Path("./tests/oVirtTinyCore64-13.11.ova")
),
os_type="win10" # optional, default "l26".
),
# A virtual machine to clone from an existing template VM.
# This is *not recommended* since it is dependent on configuring a
# customised Proxmox instance that contains the template VM before
# the eval start.
VmConfig(
vm_source_config=VmSourceConfig(
existing_vm_template_tag="java_server"
),
),
# A virtual machine that is connected to a predefined VNET.
# This is *not recommended* since it is dependent on configuring a
# customised Proxmox instance that contains SDN configurations before
# the eval start.
VmConfig(
vm_source_config=VmSourceConfig(
built_in="ubuntu24.04"
),
nics=(
VmNicConfig(
# If you reference a pre-existing VNET here, and
# set sdn_config=None in the ProxmoxSandboxEnvironmentConfig,
# it will look for the VNET alias in the existing Proxmox SDN.
vnet_alias="existing vnet alias",
),
)
),
# A virtual machine with no network access.
VmConfig(
# ... snip ...
nics=()
),
),
# You will need a separate SDN per sample, or the VMs will be able to see each other
# IP ranges *must* be distinct, unfortunately.
# If you don't care about any of this, you can set this field to the string "auto"
# and you will get an IP range somewhere in 192.168.[2 - 253].0/24
sdn_config=SdnConfig(
vnet_configs=(
VnetConfig(
# You can leave subnets blank if you are handling IPAM yourself (e.g. with your own pfsense instance as a VM)
subnets=(
SubnetConfig(
cidr=ip_network("192.168.20.0/24"),
gateway=ip_address("192.168.20.1"),
# If you set snat=False, VMs will see each other but not the wider Internet.
snat=True,
dhcp_ranges=(
DhcpRange(
start=ip_address("192.168.20.50"),
end=ip_address("192.168.20.100"),
),
),
),
),
alias="my special vnet"
),
),
# Set use_pve_ipam_dnsnmasq to True if you want your instances to be able to access the Internet
use_pve_ipam_dnsnmasq=True,
),
),
)It is recommended that you set the name= parameter for your defined VMs. This name serves two purposes:
- It will be displayed in the Proxmox web interface
- It will be the identifier you use to reference the VM in Inspect (e.g.,
sandbox("vm_name"))
You should avoid setting the same name for multiple VMs as this will cause conflicts in how Inspect references your VMs; later VMs with the same name will overwrite earlier ones in the sandbox name mapping. While both VMs would still be created in Proxmox, only the last one would be accessible through its name in Inspect. If you omit the name parameter, the VM will be registered in Inspect using its dynamically-generated ID, as vm_<id>.
Note: The (first) sandbox VM is automatically named
defaultinternally, so you can always access it withsandbox("default"), regardless of any custom name you might set for it.
By default, VMs receive IP addresses from the DHCP range specified in the subnet configuration. However, you can assign static IP addresses to VMs by specifying both a MAC address and an IPv4 address in the VmNicConfig:
nics=(
VmNicConfig(
vnet_alias="my special vnet",
mac="52:54:00:12:34:56", # Required for static IP
ipv4=ip_address("192.168.20.10") # Static IP assignment
),
)How it works:
- When both
macandipv4are specified, the system creates a DHCP static mapping (host reservation) in Proxmox - The VM will always receive the specified IP address when it boots
- The IP address must be within the subnet CIDR range but does not need to be within the DHCP range
Requirements:
- The
ipv4field requires amacaddress to be specified (validation will fail otherwise) - The IP address must fall within one of the configured subnet CIDR ranges
use_pve_ipam_dnsnmasqmust beTruein the SDN config- The Proxmox server must be patched using the patch from https://lists.proxmox.com/pipermail/pve-devel/2025-November/076472.html
sdn_config options.
If you have an existing Proxmox environment with pre-configured VNETs that you need to connect to, you can reference them by setting sdn_config=None and using the VNET aliases in your VM configurations:
sandbox = SandboxEnvironmentSpec(
type="proxmox",
config=ProxmoxSandboxEnvironmentConfig(
vms_config=(
VmConfig(
vm_source_config=VmSourceConfig(built_in="ubuntu24.04"),
nics=(
VmNicConfig(
vnet_alias="existing-vnet-alias", # Must match an existing VNET alias in Proxmox
),
),
),
),
sdn_config=None, # Disable SDN creation - use existing VNETs only
),
)Static IP address assignment is not supported with this feature.
Proxmox supports OVA import but not OVA export. It is possible to extract the disk images of VMs from a Proxmox server in qcow2 format (instructions for this can be found online).
Once you have the disk images locally, you can use the convenience script src/proxmoxsandbox/scripts/ova/convert_ova.sh
to convert it into an OVA.
This provider creates a template VM for every OVA-type VM specified in an eval. Next time you run the eval, a linked clone of the template VM will be created. This is for performance. If you change the OVA, as long as the filesize changes, a new template VM will be created. If you change the OVA but the filesize remains the same, you should manually delete it from the Proxmox server.
These template VMs are not cleaned up because that needs to happen outside the lifecycle of an Inspect eval. You need to do this manually at the moment.
Windows VMs are supported via the QEMU guest agent. To use a Windows VM:
- Create a Windows VM template on your Proxmox server with the QEMU guest agent installed and running
- Convert it to a template and tag it with
inspect;<your-tag> - Reference it in your eval config:
VmConfig(
vm_source_config=VmSourceConfig(
existing_vm_template_tag="your-tag"
),
os_type="win11", # or "win10", "win8", etc. — see schema.py for all options
uefi_boot=True,
is_sandbox=True,
ram_mb=8192,
)The os_type field determines how commands are executed inside the VM. Windows types (any value starting with w) use batch scripts instead of shell scripts. The QEMU guest agent channel on Windows is less reliable than on Linux, so transient errors are automatically retried.
Note, if you are having problems, then setting Inspect's sandbox_cleanup=False will be helpful.
If you want to log into a sandbox VM, the Proxmox UI lets you open a console window, but you might not know the password.
You can use the following command on the Proxmox server (open Datacenter -> Proxmox node -> Shell):
export PROXMOX_NODE=proxmox # change this if necessary
export VM_ID=101 # change this to the correct VM ID
export NEW_PASSWORD=Password2.0 # choose a password
export VM_USERNAME=ubuntu # change as appropriate
pvesh create "/nodes/$PROXMOX_NODE/qemu/$VM_ID/agent/exec" --command bash --command "-c" --command "echo $VM_USERNAME:$NEW_PASSWORD | chpasswd"QEMU, the virtualization library used by Proxmox, allows you to snapshot a running virtual machine, including the running processes. See snapshots.py for example tools that use this.
See ctf4.py for an example capture-the-flag eval with:
- A VM for the agent
- A victim VM which the agent must hack into and obtain the root password
Every VM created by this sandbox provider is tagged inspect.
(Tags will also be duplicated if they exist on a VM already, for existing_backup_name- and existing_vm_template_tag-type VMs)
SDN zones have the pattern [3 letters from eval task name][random 3 digits][z]. VNets are similar and can be identified from their containing zone.
Some resources will persist after the eval is complete:
- the built-in VM feature creates a template VM
inspect-ubuntu24.04 - the built-in VM feature creates a SDN zone called
inspvmz - uploaded OVAs are left in place
- cloud-init ISOs are left in place
Environment cleanup is partially implemented. There is no way to tag all the resources
created by a particular eval. Therefore the cleanup process for inspect sandbox cleanup proxmox
will delete:
- all VMs tagged
inspect - any SDN zones created with names matching the pattern above.
When using PROXMOX_CONFIG_FILE, cleanup runs against every instance in the config file.
The project follows semantic versioning and is aiming for a 1.0 release. Until then, backward-compatibility is not guaranteed.
For Linux guests, write_file payloads larger than 128 KiB are written via an ISO9660 image hot-plugged into a dedicated sata5 CD-ROM slot, sidestepping the QEMU guest-agent's ~60 KiB per-call write cap. On any failure it falls back to the chunked-QGA path, so it can only speed writes up, never break them. Windows always uses chunked QGA.
Two things worth knowing:
sata5is reserved. The slot is cold-added to everyis_sandboxVM at clone time. If yourexisting_vm_template_tagtemplate already populatessata5, the cold-add overwrites it — move that content tosata0–sata4, or disable the fast path.- Disabling it. Set
ProxmoxSandboxEnvironment.ISO_WRITE_THRESHOLD_BYTESabove your largest payload to turn it off globally. On failure it also disables itself for the affected VM and logs aWARNING; the warning site in the code lists what to check.
- Proxmox server health and config check
- Normalize having a pfSense VM as the default route for networking
- Firewall off the SDN from the Proxmox server and from other SDNs
- Support cloud-init for VM definition
- Escape hatch for Proxmox API so you can specify arbitrary parameters during VM / SDN creation
The built-in VMs (ubuntu24.04, debian13, kali2025.4) pin specific upstream image URLs in built_in_vm.py. These are not auto-updated — when a new upstream release appears (e.g. a new Kali quarterly release), the URL, the Literal type in schema.py, and all references in tests and examples must be updated together.
- Large OVA uploads use PycURL, because neither aiohttp nor httpx worked with large uploads
- Inconsistent use of task_wrapper and tenacity
See CONTRIBUTING.md