diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index 757609a456..827e1a8ba1 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -117,10 +117,18 @@ jobs: rpm-release: ${{ needs.version.outputs.rpm_release }} fedora: - name: Fedora with Rootless Podman + name: Fedora with ${{ matrix.name }} Podman needs: [build-conformance, build-rpm] runs-on: ubuntu-24.04 timeout-minutes: 45 + strategy: + fail-fast: false + matrix: + include: + - name: Rootless + configuration: podman-rootless + - name: Rootful + configuration: podman-rootful permissions: actions: read contents: read @@ -154,8 +162,10 @@ jobs: name: openshell-conformance-x86_64-unknown-linux-musl path: conformance-input - - name: Run RPM gateway continuity conformance + - name: Run RPM gateway conformance shell: bash + env: + PODMAN_CONFIGURATION: ${{ matrix.configuration }} run: | set -euo pipefail chmod +x conformance-input/openshell-conformance @@ -171,13 +181,13 @@ jobs: OPENSHELL_TEST_GUEST_CACHE_DISABLE=1 nix run .#test-guest -- \ --distro fedora \ - --with podman-rootless \ + --with "${PODMAN_CONFIGURATION}" \ --with selinux \ - --copy "${candidate_cli_package[0]}:/var/lib/openshell-conformance/candidate/openshell.rpm" \ - --copy "${candidate_gateway_package[0]}:/var/lib/openshell-conformance/candidate/openshell-gateway.rpm" \ + --install "${candidate_cli_package[0]}" \ + --install "${candidate_gateway_package[0]}" \ --copy conformance-input/openshell-conformance:/tmp/openshell-conformance \ - --copy nix/test-guest/conformance-plans/gateway-upgrade-restart.toml:/tmp/conformance-plan.toml \ - --provision openshell-rpm-latest-release \ + --copy nix/test-guest/conformance-plans/gateway-restart.toml:/tmp/conformance-plan.toml \ + --provision openshell-rpm \ --provision gateway-podman \ - --provision openshell-rpm-gateway-upgrade \ - -- /tmp/openshell-conformance run --plan /tmp/conformance-plan.toml + -- /home/openshell/.local/bin/openshell-test-guest-as-gateway-user \ + /tmp/openshell-conformance run --plan /tmp/conformance-plan.toml diff --git a/architecture/gateway.md b/architecture/gateway.md index ba325ccc2c..5f7cd701c7 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -66,6 +66,24 @@ reflection, non-callback inference APIs, and HTTP routes before normal request authentication. The operator-configured primary listener retains the full multiplexed API surface. +Rootful Podman can report a private bridge gateway before netavark assigns that +address. If an exact built-in Podman callback bind fails with +`EADDRNOTAVAIL`, the Linux gateway retries the same address with +`IP_FREEBIND`. The listener remains callback-only and becomes reachable when +the first sandbox materializes the bridge. If delayed exact binding also fails +while the gateway itself runs in a container, the gateway replaces the +loopback primary and missing bridge sockets with one IPv4 wildcard socket. The +wildcard defaults to callback-only authorization; only traffic addressed to +the configured loopback endpoint receives primary scope. This keeps user and +administrator APIs off the container's non-loopback interfaces, but it does +make the sandbox-callable gRPC surface reachable on every IPv4 interface in +that container namespace for the lifetime of the gateway process. Sandbox +mTLS/JWT authentication and the RPC allowlist remain mandatory defenses. +Delayed binding does not apply to rootless Podman, an explicit +`host_gateway_ip`, Docker or external drivers, public callback addresses, or +bind errors other than `EADDRNOTAVAIL`. A host gateway never uses the wildcard +fallback. + The `rpc_auth` classification is also the source of truth for negotiated listener exposure: marking an RPC as `sandbox` or `dual` makes it callable on these listeners. Review such changes as both authorization and network-surface diff --git a/crates/openshell-driver-podman/NETWORKING.md b/crates/openshell-driver-podman/NETWORKING.md index 567abcbfcd..d094391cba 100644 --- a/crates/openshell-driver-podman/NETWORKING.md +++ b/crates/openshell-driver-podman/NETWORKING.md @@ -281,6 +281,25 @@ the supervisor's RPCs. Otherwise, it creates an additional listener that exposes only the gateway's sandbox-callable gRPC methods. Operator, health, reflection, and HTTP requests must use the primary listener. +Netavark may not assign a rootful managed bridge gateway until the first +sandbox joins the network. If that exact private callback address fails to bind +with `EADDRNOTAVAIL`, the Linux gateway uses `IP_FREEBIND` to bind the same +address before the interface exists. The listener remains callback-only and +becomes reachable when netavark materializes the bridge. Only the in-process +built-in Podman driver can mark its discovered rootful managed-bridge address +as eligible; rootless Podman, explicit `host_gateway_ip` values, and external +drivers cannot activate delayed binding. + +If delayed exact binding fails while both the gateway and rootful Podman run +inside another Linux container, the gateway uses one scoped IPv4 wildcard +listener for that process. Connections addressed to loopback retain primary +scope; connections addressed to the Podman bridge or any other IPv4 interface +are callback-only. This exposes the sandbox-callable gRPC surface on every +IPv4 interface in the outer container namespace, so deployments should still +restrict that namespace at the container-network boundary. A gateway running +directly on a host never uses the wildcard fallback. Neither strategy exposes +operator, health, reflection, or HTTP routes on non-loopback interfaces. + ### Layer 3 Inner Sandbox Network Namespace Inside the container, the supervisor creates another network namespace for the diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index 16c5780780..daa0aef54b 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -20,8 +20,6 @@ use openshell_core::gpu::{ CdiGpuDefaultSelector, CdiGpuInventory, CdiGpuSelectionError, driver_gpu_requirements, effective_driver_gpu_count, validate_specific_gpu_device_request, }; -#[cfg(target_os = "linux")] -use openshell_core::proto::compute::v1::GatewayDefaultRouteInterfaceRequirement; #[cfg(target_os = "macos")] use openshell_core::proto::compute::v1::GatewayLoopbackInterfaceRequirement; use openshell_core::proto::compute::v1::{ @@ -29,6 +27,10 @@ use openshell_core::proto::compute::v1::{ gateway_listener_requirement::Selector, }; #[cfg(target_os = "linux")] +use openshell_core::proto::compute::v1::{ + GatewayDefaultRouteInterfaceRequirement, GatewayExactBindAddressRequirement, +}; +#[cfg(target_os = "linux")] use std::net::{IpAddr, SocketAddr}; use std::path::{Path, PathBuf}; use std::sync::Arc; @@ -586,9 +588,14 @@ impl PodmanComputeDriver { })?; Ok(vec![GatewayListenerRequirement { reason: format!("Podman network '{}' host gateway", self.config.network_name), - selector: Some(Selector::ExactBindAddress( - SocketAddr::new(gateway_ip, callback_port).to_string(), - )), + selector: Some(Selector::ExactBind(GatewayExactBindAddressRequirement { + address: SocketAddr::new(gateway_ip, callback_port).to_string(), + // A rootful managed bridge can be created after gateway + // startup. An explicit override is operator-owned, and + // rootless networking must never broaden the listener. + allow_delayed_bind: !self.rootless + && self.config.host_gateway_ip.trim().is_empty(), + })), }]) } #[cfg(target_os = "macos")] @@ -2179,7 +2186,10 @@ mod tests { assert_eq!(requirements.len(), 1); assert_eq!( requirements[0].selector, - Some(Selector::ExactBindAddress("10.89.1.1:17670".to_string())) + Some(Selector::ExactBind(GatewayExactBindAddressRequirement { + address: "10.89.1.1:17670".to_string(), + allow_delayed_bind: true, + })) ); } @@ -2199,7 +2209,10 @@ mod tests { assert_eq!( requirements[0].selector, - Some(Selector::ExactBindAddress("10.90.1.1:17670".to_string())) + Some(Selector::ExactBind(GatewayExactBindAddressRequirement { + address: "10.90.1.1:17670".to_string(), + allow_delayed_bind: false, + })) ); } diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index 2619fee5cc..54f017af8d 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -34,7 +34,7 @@ k8s-openapi = { workspace = true } # Async runtime tokio = { workspace = true } -socket2 = { workspace = true } +socket2 = { workspace = true, features = ["all"] } nix = { workspace = true } # gRPC diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 5f7524f838..004628348c 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -178,6 +178,7 @@ pub enum GatewayListenerRequirement { address: SocketAddr, driver_name: String, reason: String, + allow_delayed_bind: bool, }, DefaultRouteInterface { driver_name: String, @@ -189,6 +190,12 @@ pub enum GatewayListenerRequirement { }, } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum GatewayListenerBindPolicy { + Deny, + TrustedBuiltinPodman, +} + impl GatewayListenerRequirement { pub fn driver_name(&self) -> &str { match self { @@ -627,6 +634,7 @@ impl ComputeRuntime { driver_name: String, driver: SharedComputeDriver, driver_process: Option>, + listener_bind_policy: GatewayListenerBindPolicy, store: Arc, sandbox_index: SandboxIndex, sandbox_watch_bus: SandboxWatchBus, @@ -682,6 +690,23 @@ impl ComputeRuntime { address, driver_name: driver_name.clone(), reason: requirement.reason, + allow_delayed_bind: false, + }) + } + Selector::ExactBind(exact_bind) => { + let address = exact_bind.address.parse::().map_err(|err| { + ComputeError::Message(format!( + "compute driver '{driver_name}' returned invalid gateway listener address '{}': {err}", + exact_bind.address + )) + })?; + Ok(GatewayListenerRequirement::Exact { + address, + driver_name: driver_name.clone(), + reason: requirement.reason, + allow_delayed_bind: listener_bind_policy + == GatewayListenerBindPolicy::TrustedBuiltinPodman + && exact_bind.allow_delayed_bind, }) } Selector::DefaultRouteInterface(_) => { @@ -764,6 +789,7 @@ impl ComputeRuntime { endpoint.name, driver, endpoint.driver_process, + GatewayListenerBindPolicy::Deny, store, sandbox_index, sandbox_watch_bus, @@ -10608,6 +10634,7 @@ mod tests { "test-driver".to_string(), Arc::new(TestDriver::default()), None, + GatewayListenerBindPolicy::Deny, store, SandboxIndex::new(), SandboxWatchBus::new(), @@ -10795,6 +10822,7 @@ mod tests { address: "172.19.0.1:17670".parse().unwrap(), driver_name: "docker".to_string(), reason: "external driver managed bridge".to_string(), + allow_delayed_bind: false, }] ); diff --git a/crates/openshell-server/src/gateway_listener.rs b/crates/openshell-server/src/gateway_listener.rs index b638fc33e4..279551c8ba 100644 --- a/crates/openshell-server/src/gateway_listener.rs +++ b/crates/openshell-server/src/gateway_listener.rs @@ -4,9 +4,12 @@ use crate::compute::GatewayListenerRequirement; use openshell_core::{Error, Result}; use socket2::{Domain, Protocol, Socket, Type}; +use std::io::ErrorKind; use std::net::{IpAddr, SocketAddr}; +#[cfg(target_os = "linux")] +use std::path::Path; use tokio::net::TcpListener; -use tracing::info; +use tracing::{info, warn}; /// Authorization scope associated with a gateway listener. #[derive(Clone, Copy, Debug, Eq, PartialEq)] @@ -27,6 +30,7 @@ pub struct GatewayListenerSpec { pub scope: GatewayListenerScope, covered_addresses: Vec, provenance: Option, + allows_delayed_bind: bool, } /// Diagnostic source of a driver-requested listener. @@ -49,6 +53,7 @@ impl GatewayListenerSpec { scope, covered_addresses: Vec::new(), provenance: None, + allows_delayed_bind: false, } } @@ -177,6 +182,13 @@ fn callback_listener_spec( address: SocketAddr, requirement: &GatewayListenerRequirement, ) -> GatewayListenerSpec { + let allows_delayed_bind = matches!( + requirement, + GatewayListenerRequirement::Exact { + allow_delayed_bind: true, + .. + } + ); GatewayListenerSpec { address, scope: GatewayListenerScope::ComputeDriverCallback, @@ -185,6 +197,7 @@ fn callback_listener_spec( driver_name: requirement.driver_name().to_string(), reason: requirement.reason().to_string(), }), + allows_delayed_bind, } } @@ -272,9 +285,77 @@ pub async fn bind_gateway_listeners( ) && specs.iter().any(|candidate| { candidate.address.port() == spec.address.port() && candidate.address.is_ipv4() }); - let listener = bind_gateway_listener(spec.address, ipv6_only) - .await - .map_err(|e| Error::transport(format!("failed to bind to {}: {e}", spec.address)))?; + let listener = bind_gateway_listener(spec.address, ipv6_only).await; + let listener = match listener { + Ok(listener) => listener, + Err(err) => { + let delayed_bind_eligible = podman_delayed_bind_is_eligible(spec, err.kind()); + if delayed_bind_eligible { + match bind_gateway_listener_freebind(spec.address) { + Ok(listener) => { + let provenance = spec + .provenance + .as_ref() + .expect("delayed callback listener must include provenance"); + warn!( + address = %spec.address, + listener_purpose = "compute-driver-callback-delayed-bind", + driver = %provenance.driver_name, + reason = %provenance.reason, + bind_strategy = "linux-ip-freebind", + authorization_scope = "sandbox-callable-grpc-only", + "Podman bridge address is not available yet; gateway bound the exact callback address for delayed activation" + ); + listener + } + Err(freebind_err) => { + let Some(fallback_spec) = nested_podman_wildcard_fallback_spec( + &specs, + spec, + err.kind(), + running_in_linux_container(), + ) else { + return Err(Error::transport(format!( + "failed to bind Podman callback address {} ({err}); delayed exact bind also failed: {freebind_err}", + spec.address + ))); + }; + + // The wildcard cannot coexist with the already-bound loopback + // socket on the same port. Dropping the partial listener set is + // safe because none of it has been returned to the server yet. + drop(listeners); + let listener = bind_gateway_listener(fallback_spec.address, false) + .await + .map_err(|fallback_err| { + Error::transport(format!( + "failed to bind Podman callback address {} ({err}); delayed exact bind failed ({freebind_err}); scoped wildcard fallback {} also failed: {fallback_err}", + spec.address, fallback_spec.address + )) + })?; + let local_addr = listener.local_addr().unwrap_or(fallback_spec.address); + let fallback_spec = fallback_spec.bind_to(local_addr); + warn!( + address = %local_addr, + unavailable_callback_address = %spec.address, + listener_purpose = "nested-podman-callback-fallback", + authorization_scope = "primary-on-loopback; sandbox-callable-grpc-only-on-other-ipv4-interfaces", + "Podman bridge address is not available yet and delayed exact binding failed; gateway callback listener is exposed on all container IPv4 interfaces for this gateway process" + ); + return Ok(vec![BoundGatewayListener { + listener, + spec: fallback_spec, + }]); + } + } + } else { + return Err(Error::transport(format!( + "failed to bind to {}: {err}", + spec.address + ))); + } + } + }; let local_addr = listener.local_addr().unwrap_or(spec.address); match spec.scope { GatewayListenerScope::Primary => { @@ -308,6 +389,83 @@ pub async fn bind_gateway_listeners( Ok(listeners) } +fn nested_podman_wildcard_fallback_spec( + specs: &[GatewayListenerSpec], + failed_spec: &GatewayListenerSpec, + error_kind: ErrorKind, + running_in_container: bool, +) -> Option { + if !running_in_container + || specs.len() != 2 + || !podman_delayed_bind_is_eligible(failed_spec, error_kind) + { + return None; + } + + let primary = specs + .iter() + .find(|spec| spec.scope == GatewayListenerScope::Primary)?; + let callback = specs.iter().find(|spec| { + spec.scope == GatewayListenerScope::ComputeDriverCallback && *spec == failed_spec + })?; + let callback_provenance = callback.provenance.as_ref()?; + let (IpAddr::V4(primary_ip), IpAddr::V4(_)) = (primary.address.ip(), callback.address.ip()) + else { + return None; + }; + if !primary_ip.is_loopback() + || primary.address.port() == 0 + || primary.address.port() != callback.address.port() + { + return None; + } + + Some(GatewayListenerSpec { + address: SocketAddr::from((std::net::Ipv4Addr::UNSPECIFIED, primary.address.port())), + // The broader socket is callback-only by default. Only connections + // addressed to the original loopback endpoint retain primary scope. + scope: GatewayListenerScope::ComputeDriverCallback, + covered_addresses: vec![CoveredGatewayAddress { + address: primary.address, + scope: GatewayListenerScope::Primary, + }], + provenance: Some(callback_provenance.clone()), + allows_delayed_bind: true, + }) +} + +fn podman_delayed_bind_is_eligible( + failed_spec: &GatewayListenerSpec, + error_kind: ErrorKind, +) -> bool { + if error_kind != ErrorKind::AddrNotAvailable + || failed_spec.scope != GatewayListenerScope::ComputeDriverCallback + { + return false; + } + + let Some(callback_provenance) = failed_spec.provenance.as_ref() else { + return false; + }; + let IpAddr::V4(callback_ip) = failed_spec.address.ip() else { + return false; + }; + + callback_ip.is_private() + && failed_spec.allows_delayed_bind + && callback_provenance.driver_name == "podman" +} + +#[cfg(target_os = "linux")] +fn running_in_linux_container() -> bool { + Path::new("/.dockerenv").exists() || Path::new("/run/.containerenv").exists() +} + +#[cfg(not(target_os = "linux"))] +fn running_in_linux_container() -> bool { + false +} + fn resolve_bound_covered_addresses( covered_addresses: &[CoveredGatewayAddress], requested_listener_addr: SocketAddr, @@ -356,6 +514,32 @@ async fn bind_gateway_listener( TcpListener::bind(address).await } +#[cfg(target_os = "linux")] +fn bind_gateway_listener_freebind(address: SocketAddr) -> std::io::Result { + if !address.is_ipv4() { + return Err(std::io::Error::new( + ErrorKind::InvalidInput, + "delayed gateway listener binding requires an IPv4 address", + )); + } + let socket = Socket::new(Domain::IPV4, Type::STREAM, Some(Protocol::TCP))?; + socket.set_reuse_address(true)?; + socket.set_freebind_v4(true)?; + socket.set_nonblocking(true)?; + socket.bind(&address.into())?; + socket.listen(1024)?; + let listener: std::net::TcpListener = socket.into(); + TcpListener::from_std(listener) +} + +#[cfg(not(target_os = "linux"))] +fn bind_gateway_listener_freebind(_address: SocketAddr) -> std::io::Result { + Err(std::io::Error::new( + ErrorKind::Unsupported, + "delayed gateway listener binding is supported only on Linux", + )) +} + fn listener_covers(existing: SocketAddr, requested: SocketAddr) -> bool { if existing == requested { return true; @@ -376,9 +560,11 @@ mod tests { use super::{ GatewayListenerProvenance, GatewayListenerScope, GatewayListenerSpec, bind_gateway_listeners, gateway_listener_specs, - gateway_listener_specs_with_default_route_ip, + gateway_listener_specs_with_default_route_ip, nested_podman_wildcard_fallback_spec, + podman_delayed_bind_is_eligible, }; use crate::compute::GatewayListenerRequirement; + use std::io::ErrorKind; use std::net::SocketAddr; use std::sync::atomic::{AtomicBool, Ordering}; use tokio::net::TcpListener; @@ -435,6 +621,7 @@ mod tests { scope: GatewayListenerScope::Primary, covered_addresses: Vec::new(), provenance: None, + allows_delayed_bind: false, }, GatewayListenerSpec { address: callback, @@ -444,6 +631,7 @@ mod tests { driver_name: "alpha".to_string(), reason: "managed bridge".to_string(), }), + allows_delayed_bind: false, }, ] ); @@ -456,6 +644,7 @@ mod tests { address: "172.18.0.1:8080".parse().unwrap(), driver_name: "external-test".to_string(), reason: "external bridge".to_string(), + allow_delayed_bind: false, }; let specs = gateway_listener_specs(primary, &[requirement]).unwrap(); @@ -620,6 +809,59 @@ mod tests { assert_eq!(specs[1].scope, GatewayListenerScope::ComputeDriverCallback); } + #[test] + fn podman_delayed_bind_does_not_depend_on_primary_listener_topology() { + let callback: SocketAddr = "10.89.1.1:8080".parse().unwrap(); + + for primary in ["192.168.20.20:8080", "[::1]:8080"] { + let primary = primary.parse().unwrap(); + let specs = gateway_listener_specs(primary, &[delayed_podman_requirement(callback)]) + .expect("delayed Podman callback requirement should be valid"); + + assert!(podman_delayed_bind_is_eligible( + &specs[1], + ErrorKind::AddrNotAvailable, + )); + } + } + + #[test] + fn nested_podman_wildcard_fallback_remains_loopback_primary_only() { + let callback: SocketAddr = "10.89.1.1:8080".parse().unwrap(); + + for primary in ["192.168.20.20:8080", "[::1]:8080"] { + let primary = primary.parse().unwrap(); + let specs = gateway_listener_specs(primary, &[delayed_podman_requirement(callback)]) + .expect("delayed Podman callback requirement should be valid"); + + assert!( + nested_podman_wildcard_fallback_spec( + &specs, + &specs[1], + ErrorKind::AddrNotAvailable, + true, + ) + .is_none(), + "wildcard fallback should reject primary listener {primary}", + ); + } + + let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); + let specs = gateway_listener_specs(primary, &[delayed_podman_requirement(callback)]) + .expect("delayed Podman callback requirement should be valid"); + assert_eq!( + nested_podman_wildcard_fallback_spec( + &specs, + &specs[1], + ErrorKind::AddrNotAvailable, + true, + ) + .expect("loopback primary should permit the nested wildcard fallback") + .address, + "0.0.0.0:8080".parse().unwrap(), + ); + } + #[tokio::test] async fn failed_bind_does_not_return_partially_bound_listeners() { let occupied_listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); @@ -676,6 +918,7 @@ mod tests { address, driver_name: "alpha".to_string(), reason: "managed bridge".to_string(), + allow_delayed_bind: false, } } @@ -684,6 +927,16 @@ mod tests { address, driver_name: "beta".to_string(), reason: "managed bridge".to_string(), + allow_delayed_bind: false, + } + } + + fn delayed_podman_requirement(address: SocketAddr) -> GatewayListenerRequirement { + GatewayListenerRequirement::Exact { + address, + driver_name: "podman".to_string(), + reason: "managed bridge".to_string(), + allow_delayed_bind: true, } } @@ -707,6 +960,7 @@ mod tests { scope: GatewayListenerScope::Primary, covered_addresses: Vec::new(), provenance: None, + allows_delayed_bind: false, } } @@ -723,6 +977,7 @@ mod tests { driver_name: driver_name.to_string(), reason: reason.to_string(), }), + allows_delayed_bind: false, } } } diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index a8c8afdf08..624a31d007 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -839,11 +839,11 @@ async fn serve_gateway_listener( continue; } }; - let listener_scope = match stream.local_addr() { - Ok(local_addr) => spec.scope_for_local_addr(local_addr), + let (accepted_local_addr, listener_scope) = match stream.local_addr() { + Ok(local_addr) => (local_addr, spec.scope_for_local_addr(local_addr)), Err(e) => { debug!(error = %e, client = %addr, listen = %listen_addr, "Failed to inspect accepted local address"); - spec.scope + (listen_addr, spec.scope) } }; @@ -852,7 +852,7 @@ async fn serve_gateway_listener( spawn_gateway_connection( stream, addr, - listen_addr, + accepted_local_addr, listener_scope, service.clone(), tls_acceptor.clone(), @@ -1401,20 +1401,28 @@ async fn build_compute_runtime( }; let instance = registration.factory.build(build_context).await?; match instance { - ComputeDriverInstance::InProcess(driver) => ComputeRuntime::from_driver( - registration.name, - driver, - None, - store, - sandbox_index, - sandbox_watch_bus, - tracing_log_bus, - supervisor_sessions, - ) - .await - .map_err(|error| { - Error::execution(format!("failed to create compute runtime: {error}")) - })?, + ComputeDriverInstance::InProcess(driver) => { + let listener_bind_policy = if registration.name == "podman" { + compute::GatewayListenerBindPolicy::TrustedBuiltinPodman + } else { + compute::GatewayListenerBindPolicy::Deny + }; + ComputeRuntime::from_driver( + registration.name, + driver, + None, + listener_bind_policy, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + ) + .await + .map_err(|error| { + Error::execution(format!("failed to create compute runtime: {error}")) + })? + } ComputeDriverInstance::ManagedRemote(mut endpoint) => { endpoint.name = registration.name; ComputeRuntime::new_remote_driver( @@ -2296,6 +2304,7 @@ mod tests { address, driver_name: "docker".to_string(), reason: "managed bridge".to_string(), + allow_delayed_bind: false, } } } diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 987e66b0d9..7d8f683ccc 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -148,6 +148,24 @@ HTTP requests. A `PermissionDenied` response from an additional callback-only listener is expected for those requests. Do not broaden the primary listener to `0.0.0.0` solely to make sandbox callbacks reachable. +For rootful Podman on Linux, the bridge gateway may not exist until the first +sandbox joins. If the exact private bridge bind fails with `EADDRNOTAVAIL`, +OpenShell binds that future address with `IP_FREEBIND`. The callback-only +listener becomes reachable when netavark materializes the bridge. This applies +only to the managed bridge discovered by the built-in Podman driver; rootless +Podman, explicit `host_gateway_ip` values, and external drivers remain +ordinary exact-bind only. + +If delayed exact binding also fails while the gateway is nested inside a Linux +container, OpenShell falls back to one IPv4 wildcard socket for that gateway +process. Loopback traffic still receives primary scope; every other IPv4 +interface receives callback-only scope. The fallback therefore exposes the +sandbox-callable gRPC surface—not administrator, health, reflection, or HTTP +routes—on all IPv4 interfaces in the outer container namespace. Restrict access +to that namespace with the surrounding container network, and retain mTLS and +sandbox JWT authentication. Gateways running directly on a host never use the +wildcard fallback. + ## Docker Driver [Docker](https://www.docker.com/get-started/)-backed sandboxes run as containers on the gateway host. Use Docker for local development, single-machine gateways, and hosts that already use Docker Desktop or Docker Engine. diff --git a/nix/test-guest/README.md b/nix/test-guest/README.md index 58cf771679..d4c91f62f7 100644 --- a/nix/test-guest/README.md +++ b/nix/test-guest/README.md @@ -230,9 +230,25 @@ EOF `openshell-rpm` expects OpenShell to have been installed with `--install`. It uses the RPM-owned `/usr/bin` binaries and `openshell-gateway` user service, without copied development artifacts or a supervisor archive. Compose it with -`gateway-podman` to start the installed version. The existing upgrade flow uses -the same role with the rootless configuration before -`openshell-rpm-gateway-upgrade`. +`gateway-podman` to test the installed version directly. + +For example, run the current RPMs through rootful Podman conformance with: + +```shell +nix run .#test-guest -- \ + --distro fedora --with podman-rootful --with selinux \ + --install ./openshell.rpm \ + --install ./openshell-gateway.rpm \ + --copy ./openshell-conformance:/tmp/openshell-conformance \ + --copy nix/test-guest/conformance-plans/gateway-restart.toml:/tmp/conformance-plan.toml \ + --provision openshell-rpm \ + --provision gateway-podman \ + -- /home/openshell/.local/bin/openshell-test-guest-as-gateway-user \ + /tmp/openshell-conformance run --plan /tmp/conformance-plan.toml +``` + +Use `podman-rootless` for the equivalent rootless run. All copied binaries and +RPMs must match the guest architecture. `openshell-rpm-latest-release` downloads and installs the latest stable OpenShell GitHub release for the guest architecture, then publishes the same @@ -244,6 +260,10 @@ scenarios to the stable action-command contracts installed by provisioners. Copy the applicable plan to the guest and pass it to `openshell-conformance run --plan`. +`gateway-restart.toml` tests the installed version with smoke coverage and +sandbox continuity across a gateway restart. `gateway-upgrade-restart.toml` +retains the upgrade action contract for dedicated upgrade testing. + ## Prepared VM cache The `test-guest-cache` app ensures a prepared disk exists for one exact distro, host architecture, and ordered configuration list. It checks the local cache first, optionally pulls a matching OCI artifact, or builds and validates a new local entry on a miss: diff --git a/nix/test-guest/conformance-plans/gateway-restart.toml b/nix/test-guest/conformance-plans/gateway-restart.toml new file mode 100644 index 0000000000..2725bc80dd --- /dev/null +++ b/nix/test-guest/conformance-plans/gateway-restart.toml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +version = 1 + +[diagnostics] +command = "/home/openshell/.local/bin/openshell-test-guest-diagnostics" +timeout_secs = 60 + +[[runs]] +scenario = "smoke" + +[[runs]] +scenario = "sandbox-continuity" +workload_expectation = "reconciled" + +[[runs.actions]] +name = "gateway-restart" +command = "/home/openshell/.local/bin/openshell-test-guest-gateway-restart" +timeout_secs = 120 diff --git a/proto/compute_driver.proto b/proto/compute_driver.proto index b737e7de62..1ea7c58d22 100644 --- a/proto/compute_driver.proto +++ b/proto/compute_driver.proto @@ -111,7 +111,8 @@ message GatewayListenerRequirement { oneof selector { // Concrete IP:port address requested by the driver. The port must match - // the gateway's configured primary listener port. + // the gateway's configured primary listener port. Retained for drivers + // that predate GatewayExactBindAddressRequirement. string exact_bind_address = 2; // Ask the gateway to bind the IPv4 address selected by its default route. // This matches rootless pasta's default upstream-interface selection. @@ -119,9 +120,23 @@ message GatewayListenerRequirement { // Ask the gateway to ensure an IPv4 loopback listener is present. This // covers runtimes whose host forwarder terminates on gateway loopback. GatewayLoopbackInterfaceRequirement loopback_interface = 4; + // Concrete address with additional bind behavior requested by a trusted + // built-in driver. + GatewayExactBindAddressRequirement exact_bind = 5; } } +message GatewayExactBindAddressRequirement { + // Concrete IP:port address requested by the driver. + string address = 1; + + // Indicates that the address belongs to a driver-managed bridge that may + // not exist until the first sandbox joins it. Gateways must honor this only + // for a trusted built-in driver; external drivers cannot authorize a + // nonlocal or wildcard listener. + bool allow_delayed_bind = 2; +} + message GatewayDefaultRouteInterfaceRequirement {} message GatewayLoopbackInterfaceRequirement {} diff --git a/skills/debug-openshell-cluster/SKILL.md b/skills/debug-openshell-cluster/SKILL.md index 8e86bc0643..7a87dd219f 100644 --- a/skills/debug-openshell-cluster/SKILL.md +++ b/skills/debug-openshell-cluster/SKILL.md @@ -235,6 +235,20 @@ Common findings: - Current gateways reuse the primary listener when it covers Podman's callback address. If the primary does not cover that address, inspect the gateway startup logs for the additional callback-only listener and its provenance. +- For rootful Podman, a missing managed bridge address can trigger + `listener_purpose="compute-driver-callback-delayed-bind"`. Confirm the exact + private address belongs to the built-in driver's discovered managed bridge, + the initial bind error is `EADDRNOTAVAIL`, and the log reports + `bind_strategy="linux-ip-freebind"`. The listener is callback-only and + becomes reachable when the first sandbox materializes the bridge. +- In a nested Linux container with rootful Podman, a missing bridge address can + trigger `listener_purpose="nested-podman-callback-fallback"` only when the + delayed exact bind also fails. Confirm the failed exact address is the + built-in driver's discovered managed bridge (not rootless Podman, an explicit + `host_gateway_ip`, or an external driver), and the wildcard socket is + callback-only off loopback. The callback RPC surface is reachable on every + IPv4 interface in the outer container namespace for that gateway process, so + also inspect the surrounding container-network boundary. - Rootless slirp4netns, another named helper, or missing helper metadata requires an explicitly remote `grpc_endpoint`. An explicit `host_gateway_ip` cannot bypass slirp4netns host-loopback isolation. Do not work around @@ -680,6 +694,8 @@ configuration — check that the gateway spawned the driver binary you expect | `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` | | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | | Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route | +| Rootful Podman gateway logs `compute-driver-callback-delayed-bind` | Podman reported a private bridge gateway before netavark assigned it | Confirm the listener uses `linux-ip-freebind`; create a sandbox and verify the managed bridge acquires the logged address | +| Nested rootful Podman gateway logs `nested-podman-callback-fallback` | Both the ordinary and delayed exact binds failed before netavark assigned the bridge | Verify non-loopback destinations receive callback-only scope and restrict the outer container network | | Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected | | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | | Docker GPU sandbox fails before startup | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |