Skip to content

feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes #3088

Description

@feloy

User Story

As a platform operator running OpenShell with the MicroVM (openshell-driver-vm) compute driver on corporate workstations and hosts whose egress is only permitted through a forward proxy,
I want the sandbox supervisor to chain policy-approved egress through that proxy the same way it already does on Podman and Kubernetes,
so that agents in VM sandboxes can reach the external and internal APIs my policy allows instead of failing with timeouts and connection resets.

Problem Statement

The corporate forward proxy support added for #1792 is not reachable from the MicroVM driver.

The proxy machinery itself is driver-agnostic and already merged: openshell-supervisor-network implements HTTP CONNECT chaining, NO_PROXY matching, credential handling, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as supervisor flags. What is missing is the per-driver half: a configuration surface on the VM driver and a way for the driver to deliver those settings (and the credential and CA files they reference) into the guest.

crates/openshell-driver-vm/ contains no proxy configuration at all — grepping it for
http_proxy|https_proxy|no_proxy|proxy_auth|upstream_proxy|upstream-proxy returns nothing. The Podman driver gained this surface in #2245 and #2512, and the Kubernetes driver in #2633 (issue #2624). The VM driver was explicitly noted as the remaining gap in the investigation section of #2624 and independently reported on #1792
(comment).

The gap is larger than a config struct. On Podman and Kubernetes the driver builds the supervisor's command line directly. In the VM driver it does not: the guest init script (crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh) hardcodes exec /opt/openshell/bin/openshell-sandbox --workdir /sandbox (and the same fixed argv in exec_supervisor_in_newroot). There is no channel today for driver-owned supervisor arguments, and the supervisor's proxy flags deliberately have no environment fallback, so the existing guest environment path cannot carry them.

Impact / Why This Matters

Without this, VM-driver sandboxes on proxy-only networks have no working path to any destination that must traverse the corporate proxy, even when policy explicitly allows it.

Reported on #1792 against OpenShell v0.0.86, reproduced on three separate machines across three corporate networks: an agent dispatching gh/curl calls to an internal GitHub Enterprise host sees

Post "https://<internal-ghe-host>/api/graphql": read tcp 10.200.0.2:xxxxx->10.200.0.1:3128:
read: connection reset by peer

sometimes after a partial TLS handshake, sometimes with no bytes returned at all, while public-IP destinations from the same sandbox and session succeed. The internal host resolves into CGNAT space (100.64.0.0/10) — the corporate-DNS-interception pattern #1792 is scoped around — consistent with the supervisor dialing the destination directly rather than chaining through the proxy.

Today the only workarounds are:

  • Switch to the Podman or Kubernetes driver. Gives up the VM driver's isolation boundary, which is the reason these deployments chose it, and is not available where the host has no container runtime.
  • Set HTTPS_PROXY/NO_PROXY on the sandbox. Does not work by design. The supervisor ignores those variables and rewrites the workload's proxy environment to point at its own policy proxy.
  • Run an outboard proxy chain on the host. Duplicates the supervisor's SSRF and destination validation, keeps a second allowlist in sync, and still cannot reach guest-side TLS interception when the proxy re-signs certificates.

The result is that the driver offering the strongest isolation is the one that cannot be used in the environments that most often demand strong isolation. The failure mode is also intermittent and presents as a network flake, so it costs debugging time before it is recognized as a missing feature.

Proposed Design

Give the MicroVM driver the same operator-owned proxy surface the other first-party drivers have, reusing the merged supervisor machinery unchanged. No new proxy logic should be required; the work is driver-level configuration, guest delivery, and tests.

Configuration. Add to the [openshell.drivers.vm] gateway config section the keys the Podman driver already documents: https_proxy, no_proxy, proxy_auth_file, proxy_auth_allow_insecure, proxy_connect_by_hostname, and proxy_ca_bundle, with matching CLI flags and OPENSHELL_VM_* environment variables on the driver binary, consistent with how the driver's other settings are exposed. The upstream proxy is a property of the host's network, not of a workload, so it is deployment-level configuration and must not be settable through the per-sandbox driver_config envelope (RFC 0006 lists overriding gateway-computed config as a non-goal). Paths in proxy_auth_file and proxy_ca_bundle are host paths read by the driver.

Delivery into the guest. The settings must reach the guest supervisor as driver-owned supervisor arguments — the same flags the other drivers pass (--upstream-proxy, --upstream-no-proxy, --upstream-proxy-auth-file, --upstream-proxy-auth-allow-insecure, --upstream-proxy-connect-by-hostname, --upstream-proxy-ca-bundle) — and must not be overridable, shadowable, or forgeable from the sandbox image, the user-supplied environment, or anything else inside the guest. Because the guest init script currently execs a fixed argv, this needs a driver→supervisor argument channel that the guest treats as authoritative. The existing init drop-in manifest is the precedent to follow: the driver writes it into the overlay upperdir on every launch and the guest fails closed on anything it did not write. Both launch backends must carry it — the libkrun path (krun_set_exec with driver-built args) and the QEMU launcher path (which injects /upper/srv/openshell-env.sh into the overlay before boot).

Credential and CA material. A microVM has no bind mounts or container secrets, so the driver should stage the credential and the CA PEM into the per-sandbox guest the way it already stages the guest TLS materials and sandbox JWT (written into the overlay upperdir at fixed /opt/openshell/... paths with an explicit mode). The credential must be root-only in the guest, must never appear in the guest environment, the sandbox metadata, or logs, and must be removed with the sandbox. Note as an explicit security consideration that this places the credential at rest inside the per-sandbox overlay disk image on the host, which differs from the Podman secret model; whether that is acceptable, or whether a different in-guest delivery is needed, should be settled in review.

Validation. Fail closed, reusing the shared validators in openshell_core::driver_utils (parse_upstream_proxy_url, parse_upstream_proxy_credential, read_upstream_proxy_credential_file) that the Podman and Kubernetes drivers already call. Any present-but-invalid setting — malformed or scheme-less or port-less URL, inline user:pass@, an unreadable or malformed credential file, credentials without the insecure-auth acknowledgement, a CA bundle or auxiliary setting without a proxy URL — must fail gateway or sandbox startup with a clear error rather than silently degrading to a direct dial.

Reachability. Guest egress leaves through gvproxy, so the documented behavior needs to state where the configured proxy is expected to live and how to name it: a proxy on the host's loopback is reachable from the guest only through the gvproxy host-loopback alias (host.openshell.internal), while a proxy on the corporate network is dialed out through the host process. The gateway callback path from the guest must remain direct.

Scope. Both http:// and https:// proxy URLs, and corporate CA trust for TLS-intercepting proxies, since #2512 already merged that into the shared code — unlike #2624, which predated it.
As on the other drivers, only TLS/CONNECT egress is chained; plain-HTTP destination requests continue to dial directly. This issue does not change the VM driver's runtime network capabilities: it advertises no policy DNS or transparent TCP interception today, so proxy chaining applies to the traffic that reaches the supervisor's proxy, which is unchanged by this work.

Acceptance Criteria

  • [openshell.drivers.vm] accepts https_proxy, no_proxy, proxy_auth_file,
    proxy_auth_allow_insecure, proxy_connect_by_hostname, and proxy_ca_bundle, with
    equivalent driver CLI flags and environment variables.
  • With https_proxy configured, a policy-approved TLS destination from a VM sandbox is reached
    through a CONNECT tunnel to that proxy; the same request fails without the proxy configured on
    a proxy-only network.
  • Destinations matching no_proxy are dialed directly, including port-qualified, domain,
    wildcard, and CIDR entries, with the same semantics as the other drivers.
  • An https:// proxy URL works when its CA is supplied via proxy_ca_bundle, and a
    TLS-intercepting proxy's re-signed upstream certificates are trusted by both the supervisor
    and workloads inside the guest.
  • Proxy credentials supplied via proxy_auth_file are sent as Proxy-Authorization and never
    appear in the guest environment, sandbox metadata, driver logs, or gateway logs; they require
    proxy_auth_allow_insecure for a cleartext http:// proxy, and they are removed when the
    sandbox is deleted.
  • A sandbox cannot select, alter, or disable the upstream proxy from inside the guest —
    including via image ENV, user-supplied sandbox environment, or files placed in the image at
    the paths the driver uses.
  • Every invalid or incoherent proxy setting is fatal at gateway or sandbox startup with an error
    naming the offending key; no configuration error degrades to a direct dial.
  • A proxy reachable only on the host's loopback is usable from a VM sandbox through the
    documented host alias, and the guest→gateway callback path is unaffected.
  • An e2e case in the e2e:vm lane covers CONNECT chaining, NO_PROXY bypass, credential
    handling, https:// proxy with CA bundle, and fail-closed rejection of invalid config —
    the VM counterpart of e2e/rust/tests/podman_corporate_proxy.rs and
    kubernetes_corporate_proxy.rs.
  • docs/reference/gateway-config.mdx (MicroVM section),
    docs/reference/sandbox-compute-drivers.mdx, crates/openshell-driver-vm/README.md, and
    architecture/sandbox.md document the keys, the reachability rules, and the fail-closed
    contract.

Alternatives Considered

Honor HTTPS_PROXY/NO_PROXY from the guest environment. Rejected for the same reason as on
Podman and Kubernetes: upstream_proxy.rs ignores these deliberately, because the sandbox creator
controls that environment and the supervisor rewrites it to point workloads at the local policy
proxy. Honoring it would let a sandbox choose its own upstream or disable proxying with NO_PROXY=*.
The VM driver makes this worse, not better: build_guest_environment merges user-supplied
environment into the guest env map, so an environment-based transport here would be directly
attacker-influenced.

Per-sandbox proxy settings via template.driver_config.vm. Rejected. The upstream proxy is host
network topology, and RFC 0006 excludes overriding gateway-computed config through the caller-supplied
envelope. It would also hand sandbox creators a way to select their own egress route, which is the
opposite of the guarantee #2245 was built to provide.

A proxy or transparent redirect on the host, outside OpenShell. Rejected. It duplicates the
policy evaluation and SSRF validation the supervisor already performs, requires a second allowlist,
loses per-sandbox attribution in the OCSF event stream, and cannot supply the in-guest trust needed
for a TLS-intercepting proxy.

Bake the proxy configuration into the bootstrap image or a custom sandbox image. Rejected.
Image-resident configuration is exactly the input the design treats as untrusted, it cannot vary per
deployment without rebuilding images, and it gives no fail-closed behavior.

Do nothing; direct users to Podman or Kubernetes. Rejected. It removes the isolation boundary
these deployments selected the VM driver for, and is not an option on hosts without a container
runtime.

Agent Investigation

Findings from reading the tree at 8a13bc12:

  • The supervisor half is complete and driver-agnostic.
    crates/openshell-supervisor-network/src/upstream_proxy.rs implements CONNECT chaining,
    port- and resolution-aware NO_PROXY, CONNECT binding to validated addresses with fallback,
    https:// proxies over TLS, and corporate CA trust.
    crates/openshell-sandbox/src/main.rs:207-237 exposes --upstream-proxy,
    --upstream-no-proxy, --upstream-proxy-auth-file, --upstream-proxy-auth-allow-insecure,
    --upstream-proxy-connect-by-hostname, and --upstream-proxy-ca-bundle, with a comment stating
    these are accepted only as command-line arguments (no env =) precisely because a sandbox image
    could bake matching ENV values.
  • Shared validators exist. crates/openshell-core/src/driver_utils.rs holds
    parse_upstream_proxy_url, parse_upstream_proxy_credential,
    read_upstream_proxy_credential_file, MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES,
    UPSTREAM_PROXY_AUTH_MOUNT_PATH, and PROXY_CA_MOUNT_PATH. Both the Podman and Kubernetes
    drivers call them, so a VM driver implementation should too.
  • The VM driver has none of it. Grepping crates/openshell-driver-vm/ for
    http_proxy|https_proxy|no_proxy|proxy_auth|upstream_proxy|upstream-proxy returns no matches.
    VmDriverConfig (src/driver.rs:221) has no proxy fields, and src/main.rs has no proxy flags.
  • There is no argv channel to the guest supervisor today. scripts/openshell-vm-sandbox-init.sh
    ends with exec /opt/openshell/bin/openshell-sandbox --workdir /sandbox, and
    exec_supervisor_in_newroot() execs the same fixed argv through chroot in each of its
    candidate branches. Adding driver-owned supervisor arguments is the central piece of new
    mechanism this issue needs; the other drivers already build the supervisor command directly.
  • The guest environment is not a safe transport. build_guest_environment()
    (src/driver.rs:4459) merges user-supplied environment at lowest priority before overwriting
    driver-owned keys, and already removes sensitive keys (SANDBOX_TOKEN,
    GATEWAY_TLS_SERVER_NAME) for exactly this reason.
  • File delivery precedent exists. The driver writes per-sandbox material into the overlay
    upperdir with write_rootfs_image_file / set_rootfs_image_file_mode — guest TLS CA/cert/key at
    /opt/openshell/tls/* and the sandbox JWT at /opt/openshell/auth/sandbox.jwt
    (openshell_core::container_paths::VM_GUEST_*). The same path can carry the proxy credential and
    CA bundle. The fail-closed pattern to copy is the init drop-in manifest
    (VM_GUEST_INIT_DROPIN_MANIFEST): written every launch, and the guest runs only what the
    manifest lists, so an image cannot smuggle entries in.
  • Two launch backends. libkrun sets exec path, args, and env via set_exec
    (src/runtime.rs:837); the QEMU launcher has no krun_set_exec equivalent and instead injects
    /upper/srv/openshell-env.sh into the overlay (write_guest_env_file, src/runtime.rs:259).
    Both need to carry the new settings.
  • Egress topology. Guest traffic leaves through gvproxy; GVPROXY_HOST_LOOPBACK_IP
    (192.168.127.254) NATs to the host's 127.0.0.1 and is aliased as host.openshell.internal,
    seeded into the guest's /etc/hosts by the init script. A host-local proxy is only reachable
    through that alias.
  • VM runtime capabilities. build_guest_environment sets
    NETWORK_RUNTIME_CAPABILITIES to empty for the VM driver, with the comment that it does not yet
    provide policy DNS and transparent TCP interception. This bounds what traffic reaches the
    supervisor proxy at all, and is unchanged by this issue.
  • CI lane exists. e2e:vm runs through .github/workflows/e2e-vm-test.yml (invoked from
    branch-e2e.yml, release-dev.yml, release-tag.yml), so a vm_corporate_proxy.rs e2e test has
    a home alongside podman_corporate_proxy.rs and kubernetes_corporate_proxy.rs.
  • Docs surface. The MicroVM section of docs/reference/gateway-config.mdx (around line 766)
    lists the [openshell.drivers.vm] keys and currently has no proxy entries, unlike the Podman
    section directly above it.

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions