diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 45e95234b1..d1932283bd 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -178,6 +178,13 @@ Common findings: - A workdir rejected as a special filesystem or OpenShell control-path collision cannot be made valid with permissions. Move the image workdir away from kernel-backed mounts and the concrete supervisor, TLS, token, runtime, and socket paths named in the error. - Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`. - Sandbox never registers: check gateway logs and supervisor callback endpoint. +- On macOS, repeated `Policy fetch failed after 5 attempts` messages with a + Homebrew gateway bound to `[::1]:17670` indicate that the Docker + `host-gateway` IPv4 route has no matching callback listener. Current releases + leave `bind_address` unset in the Homebrew config, use the built-in + `127.0.0.1:17670` primary listener, and reuse it for authenticated sandbox + callbacks. On an older release, set `bind_address = "127.0.0.1:17670"` or + upgrade. - Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`. - `mise run e2e:docker:gpu` fails with `docker info --format json did not report any discovered NVIDIA CDI GPU devices`: Docker may report `CDISpecDirs` while still having no generated NVIDIA CDI specs. Verify `.DiscoveredDevices` contains entries such as `nvidia.com/gpu=all`, verify `/etc/cdi` or `/var/run/cdi` contains a generated NVIDIA spec, and check that `nvidia-cdi-refresh.service` and `nvidia-cdi-refresh.path` from NVIDIA Container Toolkit are enabled and healthy. The service is a one-shot unit, so `inactive (dead)` can be normal after a successful run; use `systemctl status` and `journalctl` to distinguish success from a skipped or failed refresh. NVIDIA recommends enabling the path and service units, and restarting `nvidia-cdi-refresh.service` to regenerate missing or stale CDI specs. If specs are generated but Docker still reports no discovered devices, restart Docker or reload the daemon and re-check `docker info`. @@ -207,14 +214,9 @@ Common findings: error: inspect `podman info --debug`, the configured Podman network, and the host's IPv4 default route. Rootless pasta uses the private source address selected by that route; rootful Podman uses the bridge gateway address. -- Callback discovery reports that the requested address equals the primary - listener: configure a distinct primary address. For Podman Machine, bind the - primary listener to IPv6 loopback, for example - `bind_address = "[::1]:17670"`, and register the CLI endpoint as - `https://localhost:17670`. The generated certificate includes `localhost`, - while a raw `https://[::1]:17670` endpoint can fail TLS setup with - `invalid dns name`. This leaves `127.0.0.1:17670` available for the - callback-only listener. +- Current gateways reuse the primary listener when it covers Podman's callback + address. If the primary does not cover that address, inspect the gateway + startup logs for the additional callback-only listener and its provenance. - Rootless slirp4netns, another named helper, or missing helper metadata requires an explicitly remote `grpc_endpoint`. An explicit `host_gateway_ip` cannot bypass slirp4netns host-loopback isolation. Do not work around @@ -448,7 +450,7 @@ openshell logs | `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs | | Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs | | Gateway exits while resolving compute-driver listener requirements | Callback alias topology is unsupported, the Podman network cannot be inspected, or the selected address is not private/authorized | Gateway startup error, `podman info --debug`, Podman network inspection, host IPv4 default route | -| Admin, health, reflection, or HTTP request is denied on a Docker/Podman callback address | Negotiated callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected | +| Admin, health, reflection, or HTTP request is denied on an additional Docker/Podman callback-only listener | Additional callback listeners intentionally expose only sandbox-callable gRPC methods | Retry through the gateway's primary endpoint; inspect the listener-purpose startup log if the address was unexpected | | Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs | | Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` | | Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod ` | diff --git a/architecture/gateway.md b/architecture/gateway.md index cac4f91e6b..a9831ae220 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -37,19 +37,17 @@ health, metrics, or tunnel routes. The plaintext service router also rejects browser requests whose Fetch Metadata, Origin, or Referer headers indicate a cross-origin or sibling-subdomain request. -Docker and Podman may negotiate additional listeners that make the gateway -reachable from their local sandbox network topology. Those listeners accept +Docker and Podman report the local address through which their sandboxes can +reach the gateway. When the primary listener covers that address, the gateway +reuses it; sandbox JWT authentication and its RPC allowlist remain the callback +authorization boundary. When the primary listener does not cover the address, +the gateway adds a callback-only listener. Additional callback listeners accept only gRPC methods classified as sandbox-callable by the gateway's generated authorization metadata. They reject user and administrator APIs, health, reflection, non-callback inference APIs, and HTTP routes before normal request authentication. The operator-configured primary listener retains the full multiplexed API surface. -The gateway rejects a callback requirement that resolves to the exact primary -listener address because one socket cannot preserve two authorization scopes. -A wildcard primary listener may cover a callback address because the accepted -connection's concrete local address still selects the callback-only scope. - The `rpc_auth` classification is also the source of truth for negotiated listener exposure: marking an RPC as `sandbox` or `dual` makes it callable on these listeners. Review such changes as both authorization and network-surface @@ -699,8 +697,10 @@ system entry instead of pretending to delete package-manager owned state. - Compute runtimes own the mechanics of starting workloads and injecting callback configuration. - Docker-backed local gateways use Docker's `host-gateway` callback alias on - macOS and Docker Desktop-style runtimes. Native Linux Docker may expose an - additional bridge-gateway listener because the host can bind that bridge IP. + macOS and Docker Desktop-style runtimes. They request IPv4 loopback callback + reachability and add a listener only when the primary does not cover it. + Native Linux Docker may expose an additional bridge-gateway listener because + the host can bind that bridge IP. - Podman-backed macOS gateways use gvproxy's host-loopback IP for sandbox host aliases by default so stale Podman machine images do not need Podman's `host-gateway` resolver. Linux Podman keeps the resolver unless diff --git a/crates/openshell-cli/src/tls.rs b/crates/openshell-cli/src/tls.rs index 2eadafc71a..c24b84c7dc 100644 --- a/crates/openshell-cli/src/tls.rs +++ b/crates/openshell-cli/src/tls.rs @@ -22,6 +22,7 @@ use tokio::sync::Mutex; use tonic::service::interceptor::InterceptedService; use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity}; use tracing::debug; +use url::{Host, Url}; /// Concrete gRPC client type used by all commands. pub type GrpcClient = OpenShellClient>; @@ -223,12 +224,33 @@ pub fn build_rustls_config(materials: &TlsMaterials) -> Result ClientTlsConfig { +pub fn build_tonic_tls_config(server: &str, materials: &TlsMaterials) -> Result { let ca_cert = Certificate::from_pem(materials.ca.clone()); let identity = Identity::from_pem(materials.cert.clone(), materials.key.clone()); - ClientTlsConfig::new() - .ca_certificate(ca_cert) - .identity(identity) + tls_config_with_server_name( + server, + ClientTlsConfig::new() + .ca_certificate(ca_cert) + .identity(identity), + ) +} + +fn tls_config_with_server_name(server: &str, config: ClientTlsConfig) -> Result { + Ok(config.domain_name(tls_server_name(server)?)) +} + +fn tls_server_name(server: &str) -> Result { + let endpoint = Url::parse(server) + .into_diagnostic() + .wrap_err_with(|| format!("invalid gateway endpoint '{server}'"))?; + let host = endpoint + .host() + .ok_or_else(|| miette::miette!("gateway endpoint '{server}' has no host"))?; + Ok(match host { + Host::Domain(domain) => domain.to_string(), + Host::Ipv4(address) => address.to_string(), + Host::Ipv6(address) => address.to_string(), + }) } #[derive(Debug)] @@ -402,22 +424,20 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result { // Bearer auth over HTTPS: use mTLS certs for the transport layer when // available (server may still require client certs), and layer the // Bearer token on top via the interceptor. - require_tls_materials(server, tls).map_or_else( - |_| { - let resolved = tls.with_default_paths(server); - resolved - .ca - .as_ref() - .and_then(|ca_path| std::fs::read(ca_path).ok()) - .map_or_else( - || ClientTlsConfig::new().with_enabled_roots(), - |ca_pem| { - ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)) - }, - ) - }, - |materials| build_tonic_tls_config(&materials), - ) + if let Ok(materials) = require_tls_materials(server, tls) { + build_tonic_tls_config(server, &materials)? + } else { + let resolved = tls.with_default_paths(server); + let config = resolved + .ca + .as_ref() + .and_then(|ca_path| std::fs::read(ca_path).ok()) + .map_or_else( + || ClientTlsConfig::new().with_enabled_roots(), + |ca_pem| ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem)), + ); + tls_config_with_server_name(server, config)? + } } else if tls.edge_token.is_some() { // Edge bearer mode — routed through tunnel above; if we reach here // the server is not HTTPS so connect plaintext. @@ -425,7 +445,7 @@ pub async fn build_channel(server: &str, tls: &TlsOptions) -> Result { } else { // Standard mTLS: private CA + client cert. let materials = require_tls_materials(server, tls)?; - build_tonic_tls_config(&materials) + build_tonic_tls_config(server, &materials)? }; endpoint = endpoint.tls_config(tls_config).into_diagnostic()?; endpoint.connect().await.into_diagnostic() @@ -451,3 +471,25 @@ pub async fn grpc_inference_client(server: &str, tls: &TlsOptions) -> Result, ssh_socket_path: String, stop_timeout_secs: u32, log_level: String, @@ -428,6 +429,8 @@ impl DockerComputeDriver { let host_gateway_ip = parse_optional_host_gateway_ip(&docker_config.host_gateway_ip)?; let gateway_route = docker_gateway_route(&info, bridge_gateway_ip, gateway_port, host_gateway_ip); + let gateway_callback_bind_address = + docker_gateway_callback_bind_address(&gateway_route, config.bind_address); let mut docker_config = docker_config.clone(); if docker_config.grpc_endpoint.trim().is_empty() { let scheme = if docker_guest_tls_configured(&docker_config) { @@ -456,6 +459,7 @@ impl DockerComputeDriver { grpc_endpoint, network_name, gateway_route, + gateway_callback_bind_address, ssh_socket_path: docker_config.ssh_socket_path.clone(), stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, log_level: config.log_level.clone(), @@ -1577,15 +1581,19 @@ impl ComputeDriver for DockerComputeDriver { &self, _request: Request, ) -> Result, Status> { - let requirements = match self.config.gateway_route { - DockerGatewayRoute::Bridge { bind_address, .. } => { - vec![GatewayListenerRequirement { - reason: "docker managed bridge gateway".to_string(), - selector: Some(Selector::ExactBindAddress(bind_address.to_string())), - }] - } - DockerGatewayRoute::HostGateway => Vec::new(), - }; + let requirements = + self.config + .gateway_callback_bind_address + .map_or_else(Vec::new, |bind_address| { + vec![GatewayListenerRequirement { + reason: match self.config.gateway_route { + DockerGatewayRoute::Bridge { .. } => "docker managed bridge gateway", + DockerGatewayRoute::HostGateway => "docker host-gateway IPv4 loopback", + } + .to_string(), + selector: Some(Selector::ExactBindAddress(bind_address.to_string())), + }] + }); Ok(Response::new(GetGatewayListenerRequirementsResponse { requirements, })) @@ -2788,6 +2796,22 @@ fn docker_gateway_route_for_host( } } +fn docker_gateway_callback_bind_address( + route: &DockerGatewayRoute, + primary_bind_address: SocketAddr, +) -> Option { + match route { + DockerGatewayRoute::Bridge { bind_address, .. } => Some(*bind_address), + DockerGatewayRoute::HostGateway => match primary_bind_address.ip() { + IpAddr::V4(ip) if ip.is_unspecified() || ip == Ipv4Addr::LOCALHOST => None, + _ => Some(SocketAddr::new( + IpAddr::V4(Ipv4Addr::LOCALHOST), + primary_bind_address.port(), + )), + }, + } +} + fn host_runtime_requires_host_gateway_alias() -> bool { cfg!(target_os = "macos") } diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index 13c987235a..3d209aa7c2 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -102,6 +102,10 @@ fn runtime_config() -> DockerDriverRuntimeConfig { ), host_alias_ip: IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), }, + gateway_callback_bind_address: Some(SocketAddr::new( + IpAddr::V4(Ipv4Addr::new(172, 18, 0, 1)), + DEFAULT_SERVER_PORT, + )), ssh_socket_path: "/run/openshell/ssh.sock".to_string(), stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, log_level: "info".to_string(), @@ -186,6 +190,7 @@ async fn gateway_listener_requirements_report_managed_bridge_address() { async fn gateway_listener_requirements_are_empty_for_host_gateway_route() { let mut config = runtime_config(); config.gateway_route = DockerGatewayRoute::HostGateway; + config.gateway_callback_bind_address = None; let driver = test_driver_with_config(config); let response = driver @@ -197,6 +202,26 @@ async fn gateway_listener_requirements_are_empty_for_host_gateway_route() { assert!(response.requirements.is_empty()); } +#[tokio::test] +async fn host_gateway_route_reports_ipv4_loopback_callback_listener() { + let mut config = runtime_config(); + config.gateway_route = DockerGatewayRoute::HostGateway; + config.gateway_callback_bind_address = Some("127.0.0.1:17670".parse().unwrap()); + let driver = test_driver_with_config(config); + + let response = driver + .get_gateway_listener_requirements(Request::new(GetGatewayListenerRequirementsRequest {})) + .await + .unwrap() + .into_inner(); + + assert_eq!(response.requirements.len(), 1); + assert_eq!( + response.requirements[0].selector, + Some(Selector::ExactBindAddress("127.0.0.1:17670".to_string())) + ); +} + #[test] fn container_visible_endpoint_rewrites_loopback_hosts() { assert_eq!( @@ -298,6 +323,31 @@ fn docker_gateway_route_uses_host_gateway_for_docker_desktop() { ); } +#[test] +fn host_gateway_route_requests_ipv4_loopback_for_ipv6_primary() { + assert_eq!( + docker_gateway_callback_bind_address( + &DockerGatewayRoute::HostGateway, + "[::1]:17670".parse().unwrap(), + ), + Some("127.0.0.1:17670".parse().unwrap()) + ); +} + +#[test] +fn host_gateway_route_reuses_ipv4_primary_when_it_covers_loopback() { + for primary in ["127.0.0.1:17670", "0.0.0.0:17670"] { + assert_eq!( + docker_gateway_callback_bind_address( + &DockerGatewayRoute::HostGateway, + primary.parse().unwrap(), + ), + None, + "{primary} already covers the IPv4 loopback callback" + ); + } +} + #[test] fn docker_gateway_route_uses_host_gateway_for_colima() { let info = SystemInfo { diff --git a/crates/openshell-driver-podman/NETWORKING.md b/crates/openshell-driver-podman/NETWORKING.md index 4b2ae7ff29..843cb0907d 100644 --- a/crates/openshell-driver-podman/NETWORKING.md +++ b/crates/openshell-driver-podman/NETWORKING.md @@ -275,13 +275,11 @@ and rootless pasta because the driver maps both local callback aliases to that literal. Other rootless helpers still fail closed. Podman Machine requests gateway loopback because its configured address is guest-visible and gvproxy terminates that route on host loopback. The gateway validates and binds every -accepted callback listener. A callback address cannot equal the exact primary -listener address because the gateway could not distinguish their authorization -scopes. In particular, a Podman Machine gateway using the IPv4 loopback -callback must place its primary listener on another address, such as IPv6 -loopback (`[::1]:17670`). Negotiated callback listeners expose only the -gateway's sandbox-callable gRPC methods. Operator, health, reflection, and HTTP -requests must use the primary listener. +accepted callback requirement. If the primary listener covers the requested +address, the gateway reuses it and relies on sandbox JWT authorization to limit +the supervisor's RPCs. Otherwise, it creates an additional listener that +exposes only the gateway's sandbox-callable gRPC methods. Operator, health, +reflection, and HTTP requests must use the primary listener. ### Layer 3 Inner Sandbox Network Namespace diff --git a/crates/openshell-server/src/gateway_listener.rs b/crates/openshell-server/src/gateway_listener.rs index b42069d848..4e4c7e80d0 100644 --- a/crates/openshell-server/src/gateway_listener.rs +++ b/crates/openshell-server/src/gateway_listener.rs @@ -103,7 +103,7 @@ fn gateway_listener_specs_with_default_route_ip( continue; }; validate_gateway_listener_requirement(bind_address, requirement)?; - add_callback_listener_spec(&mut specs, *address, requirement)?; + add_callback_listener_spec(&mut specs, *address, requirement); } for requirement in requirements { @@ -127,7 +127,7 @@ fn gateway_listener_specs_with_default_route_ip( } let address = SocketAddr::new(ip, bind_address.port()); validate_resolved_gateway_listener(bind_address, address)?; - add_callback_listener_spec(&mut specs, address, requirement)?; + add_callback_listener_spec(&mut specs, address, requirement); } for requirement in requirements { @@ -137,7 +137,7 @@ fn gateway_listener_specs_with_default_route_ip( validate_gateway_listener_requirement(bind_address, requirement)?; let address = SocketAddr::from(([127, 0, 0, 1], bind_address.port())); validate_resolved_gateway_listener(bind_address, address)?; - add_callback_listener_spec(&mut specs, address, requirement)?; + add_callback_listener_spec(&mut specs, address, requirement); } Ok(specs) @@ -147,20 +147,17 @@ fn add_callback_listener_spec( specs: &mut Vec, address: SocketAddr, requirement: &GatewayListenerRequirement, -) -> Result<()> { +) { let scope = GatewayListenerScope::ComputeDriverCallback; if let Some(existing) = specs .iter_mut() .find(|existing| listener_covers(existing.address, address)) { + if existing.scope == GatewayListenerScope::Primary { + return; + } if existing.address == address { - if existing.scope == GatewayListenerScope::Primary { - return Err(Error::config(format!( - "compute driver '{}' requested gateway callback listener {address}, but it is the same address as the primary listener; callback-only authorization cannot be preserved", - requirement.driver_name() - ))); - } - return Ok(()); + return; } if !existing .covered_addresses @@ -171,10 +168,9 @@ fn add_callback_listener_spec( .covered_addresses .push(CoveredGatewayAddress { address, scope }); } - return Ok(()); + return; } specs.push(callback_listener_spec(address, requirement)); - Ok(()) } fn callback_listener_spec( @@ -392,8 +388,8 @@ fn listener_covers(existing: SocketAddr, requested: SocketAddr) -> bool { #[cfg(test)] mod tests { use super::{ - CoveredGatewayAddress, GatewayListenerProvenance, GatewayListenerScope, - GatewayListenerSpec, bind_gateway_listeners, gateway_listener_specs, + GatewayListenerProvenance, GatewayListenerScope, GatewayListenerSpec, + bind_gateway_listeners, gateway_listener_specs, gateway_listener_specs_with_default_route_ip, }; use crate::compute::GatewayListenerRequirement; @@ -402,7 +398,7 @@ mod tests { use tokio::net::TcpListener; #[test] - fn gateway_listener_specs_track_driver_address_covered_by_wildcard() { + fn gateway_listener_specs_reuse_primary_when_wildcard_covers_driver_address() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); let requirements = [ @@ -412,20 +408,12 @@ mod tests { assert_eq!( gateway_listener_specs(primary, &requirements).unwrap(), - vec![GatewayListenerSpec { - address: primary, - scope: GatewayListenerScope::Primary, - covered_addresses: vec![CoveredGatewayAddress { - address: docker, - scope: GatewayListenerScope::ComputeDriverCallback, - }], - provenance: None, - }] + vec![primary_listener_spec(primary)] ); } #[test] - fn gateway_listener_scope_for_local_addr_uses_covered_address_scope() { + fn gateway_listener_scope_for_reused_primary_remains_primary() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); let loopback: SocketAddr = "127.0.0.1:8080".parse().unwrap(); @@ -436,7 +424,7 @@ mod tests { assert_eq!( spec.scope_for_local_addr(docker), - GatewayListenerScope::ComputeDriverCallback, + GatewayListenerScope::Primary, ); assert_eq!( spec.scope_for_local_addr(loopback), @@ -521,14 +509,14 @@ mod tests { } #[test] - fn gateway_listener_specs_track_podman_exact_when_primary_covers_it() { + fn gateway_listener_specs_reuse_primary_when_it_covers_podman_exact() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); let podman_gateway: SocketAddr = "10.89.1.1:8080".parse().unwrap(); assert_eq!( gateway_listener_specs(primary, &[podman_listener_requirement(podman_gateway)],) .unwrap(), - vec![primary_listener_spec_with_covered(primary, podman_gateway,)] + vec![primary_listener_spec(primary)] ); } @@ -570,11 +558,9 @@ mod tests { } #[test] - fn gateway_listener_specs_track_default_route_when_primary_is_ipv4_wildcard() { + fn gateway_listener_specs_reuse_ipv4_wildcard_for_default_route() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); let default_route_ip = "192.168.20.20".parse().unwrap(); - let callback = "192.168.20.20:8080".parse().unwrap(); - assert_eq!( gateway_listener_specs_with_default_route_ip( primary, @@ -582,7 +568,7 @@ mod tests { Some(default_route_ip), ) .unwrap(), - vec![primary_listener_spec_with_covered(primary, callback)] + vec![primary_listener_spec(primary)] ); } @@ -604,28 +590,23 @@ mod tests { } #[test] - fn gateway_listener_specs_track_podman_loopback_when_wildcard_primary_covers_it() { + fn gateway_listener_specs_reuse_wildcard_primary_for_podman_loopback() { let primary = "0.0.0.0:8080".parse().unwrap(); - let loopback = "127.0.0.1:8080".parse().unwrap(); assert_eq!( gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(), - vec![primary_listener_spec_with_covered(primary, loopback)] + vec![primary_listener_spec(primary)] ); } #[test] - fn gateway_listener_specs_reject_callback_matching_primary_address() { + fn gateway_listener_specs_reuse_matching_primary_address() { let primary = "127.0.0.1:8080".parse().unwrap(); - let err = - gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap_err(); - - assert!( - err.to_string() - .contains("same address as the primary listener") + assert_eq!( + gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(), + vec![primary_listener_spec(primary)] ); - assert!(err.to_string().contains("callback-only authorization")); } #[test] @@ -742,21 +723,6 @@ mod tests { } } - fn primary_listener_spec_with_covered( - address: SocketAddr, - covered_address: SocketAddr, - ) -> GatewayListenerSpec { - GatewayListenerSpec { - address, - scope: GatewayListenerScope::Primary, - covered_addresses: vec![CoveredGatewayAddress { - address: covered_address, - scope: GatewayListenerScope::ComputeDriverCallback, - }], - provenance: None, - } - } - fn callback_listener_spec( address: SocketAddr, driver_name: &str, diff --git a/docs/about/installation.mdx b/docs/about/installation.mdx index 58cec98f37..e464d3da42 100644 --- a/docs/about/installation.mdx +++ b/docs/about/installation.mdx @@ -38,7 +38,7 @@ For detailed driver behavior, refer to [Sandbox Compute Drivers](/reference/sand On macOS, the install script uses Homebrew. The Homebrew package installs the `openshell` CLI, the gateway binary, and a Homebrew-managed gateway service. -The Homebrew service listens on `https://[::1]:17670` and generates a local mTLS bundle on install. The formula creates a Homebrew prefix config, such as `/opt/homebrew/var/openshell/gateway.toml`, with this IPv6 loopback default so Podman can use its separate IPv4 loopback callback listener. The gateway reads `~/.config/openshell/gateway.toml` instead when that file exists. Homebrew preserves existing prefix and user configs during upgrades. +The Homebrew service uses the gateway's built-in `127.0.0.1:17670` listener and generates a local mTLS bundle on install. The installer registers `https://localhost:17670` with the CLI so TLS uses a DNS name covered by the generated certificate. The formula creates a Homebrew prefix config, such as `/opt/homebrew/var/openshell/gateway.toml`, without overriding `bind_address`. Docker Desktop and Podman Machine reuse the primary listener for sandbox callbacks when they can reach it. The gateway reads `~/.config/openshell/gateway.toml` instead when that file exists. Homebrew preserves user-edited prefix and user configs during upgrades; it removes the IPv6 bind only from an unchanged config generated by the affected formula. The CLI reads the client bundle from `~/.config/openshell/gateways/openshell/mtls/`. diff --git a/docs/reference/gateway-auth.mdx b/docs/reference/gateway-auth.mdx index 2ef54de70c..1d318f109f 100644 --- a/docs/reference/gateway-auth.mdx +++ b/docs/reference/gateway-auth.mdx @@ -47,7 +47,7 @@ Set these environment variables before starting the gateway: For local access, the server certificate must be valid for the endpoint the CLI uses. Include `localhost`, `127.0.0.1`, and `::1` in the certificate SANs when users connect to a local gateway through loopback. -Package-managed local gateways generate this bundle automatically for the `openshell` gateway name. Homebrew uses `https://[::1]:17670` by default; Debian and RPM use `https://127.0.0.1:17670`. +Package-managed local gateways generate this bundle automatically for the `openshell` gateway name. Homebrew registers `https://localhost:17670`; Debian and RPM use `https://127.0.0.1:17670`. When you register a package-managed local gateway with `openshell gateway add --local --name openshell`, the CLI refreshes its mTLS bundle from the package-managed TLS directory. On Homebrew, the gateway service also mirrors the Docker sandbox client bundle into `$HOME/.local/state/openshell/homebrew/tls` before startup so Docker Desktop can bind-mount the files into sandbox containers. diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 2cd10b8a0b..1c0d5e9ef1 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -31,7 +31,7 @@ Package-managed gateways do not require a TOML file. Create one at the package's The Fedora/RHEL RPM template leaves `[openshell.gateway].bind_address` unset. The gateway therefore uses its built-in `127.0.0.1:17670` primary listener. The Podman driver negotiates separate, restricted listeners for sandbox callbacks, so the primary listener does not need a wildcard address. Set `bind_address` explicitly only when clients must reach the primary multiplexed API through another interface. -The Homebrew formula creates its prefix config once with `bind_address = "[::1]:17670"`. Keeping the primary API on IPv6 loopback leaves IPv4 loopback available for Podman Machine's sandbox callback listener. A user config takes precedence, and upgrades do not overwrite either config. +The Homebrew formula creates its prefix config without setting `bind_address`, so the gateway uses its built-in `127.0.0.1:17670` primary listener. Docker Desktop and Podman Machine reuse that listener for sandbox callbacks. A user config takes precedence. Upgrades preserve user-edited configs and migrate only an unchanged prefix config generated with the affected IPv6-loopback default. ## Layout diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index b05bf05581..c7e1a50ed2 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -113,17 +113,15 @@ It overrides the gateway's configured default runtime class for that sandbox, while a typed `SandboxTemplate.runtime_class_name` value from the API still takes precedence. -Docker and Podman callback listeners accept only supervisor callback gRPC -methods. Use the gateway's primary endpoint for CLI, administrator, health, -reflection, inference-route management, and HTTP requests. A -`PermissionDenied` response from one of the sandbox-visible callback addresses -is expected for those requests. The gateway fails startup if a callback -requirement resolves to the exact primary listener address because one socket -cannot preserve both authorization scopes. For the IPv4-loopback callback used -by Podman Machine, set `bind_address = "[::1]:17670"` for the primary listener -and register `https://localhost:17670` as the CLI endpoint. The hostname matches -the generated certificate and avoids the TLS transport error produced by a raw -IPv6-literal endpoint. Do not broaden the primary listener to `0.0.0.0`. +Docker and Podman report the address through which their sandboxes can reach +the gateway. If the primary listener covers that address, the gateway reuses +it and sandbox JWT authentication restricts the supervisor to its callback RPC +allowlist. If the primary listener is not reachable through that address, the +gateway creates an additional callback-only listener. Use the primary endpoint +for CLI, administrator, health, reflection, inference-route management, and +HTTP requests. A `PermissionDenied` response from an additional callback-only +listener is expected for those requests. Do not broaden the primary listener +to `0.0.0.0` solely to make sandbox callbacks reachable. ## Docker Driver @@ -131,6 +129,11 @@ IPv6-literal endpoint. Do not broaden the primary listener to `0.0.0.0`. The gateway talks to the Docker daemon to create sandbox containers. Docker is also required for local image builds from directories or Dockerfiles. +Docker Desktop and compatible macOS runtimes route `host.openshell.internal` +through an IPv4 host-gateway alias. The gateway reuses an IPv4 primary listener +that already covers loopback. Otherwise, the Docker driver requests a separate +`127.0.0.1:` callback-only listener. + For maintainer-level implementation details, refer to the [Docker driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-docker/README.md). Select Docker with `compute_drivers = ["docker"]` in `[openshell.gateway]`. Configure Docker driver values such as `socket_path`, `grpc_endpoint`, `network_name`, `supervisor_bin`, `supervisor_image`, `image_pull_policy`, `ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.docker]`. When `socket_path` is unset, the driver uses the same responsive local socket selected by auto-detection. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds. diff --git a/install.sh b/install.sh index a623cd14b4..faa10c3e3b 100755 --- a/install.sh +++ b/install.sh @@ -470,7 +470,7 @@ detect_platform() { local_gateway_endpoint() { case "${PLATFORM:-$(detect_platform)}" in darwin) - printf 'https://[::1]:%s\n' "$LOCAL_GATEWAY_PORT" + printf 'https://localhost:%s\n' "$LOCAL_GATEWAY_PORT" ;; *) printf 'https://127.0.0.1:%s\n' "$LOCAL_GATEWAY_PORT" diff --git a/python/openshell/release_formula_test.py b/python/openshell/release_formula_test.py index d22705afa3..f6a1f7d13a 100644 --- a/python/openshell/release_formula_test.py +++ b/python/openshell/release_formula_test.py @@ -3,6 +3,7 @@ from __future__ import annotations +import re import subprocess import sys from pathlib import Path @@ -54,13 +55,23 @@ def test_generate_homebrew_formula_uses_tagged_macos_driver_asset_without_defaul assert "OPENSHELL_DRIVERS: " not in formula assert 'OPENSHELL_GATEWAY_CONFIG: "#{var}/openshell/gateway.toml"' not in formula assert "init-gateway-config.sh" not in formula - assert 'bind_address = "127.0.0.1:17670"' not in formula assert 'gateway_config = var/"openshell/gateway.toml"' in formula assert "unless gateway_config.exist?" in formula + generated_config = re.search( + r"gateway_config_contents = <<~TOML\n(?P.*?)\n TOML", + formula, + flags=re.DOTALL, + ) + assert generated_config is not None + assert "[openshell.gateway]" in generated_config.group("contents") + assert "bind_address =" not in generated_config.group("contents") assert 'bind_address = "[::1]:17670"' in formula + assert "gateway_config.read == legacy_ipv6_gateway_config_contents" in formula + assert "gateway_config.write gateway_config_contents" in formula assert '# compute_drivers = ["vm"]' not in formula assert ( - "openshell gateway add https://[::1]:17670 --local --name openshell" in formula + "openshell gateway add https://localhost:17670 --local --name openshell" + in formula ) assert 'run opt_libexec/"openshell-gateway-homebrew-service"' in formula assert 'xdg_config_home="${XDG_CONFIG_HOME:-${HOME}/.config}"' in formula diff --git a/tasks/scripts/release.py b/tasks/scripts/release.py index 243c72e8ee..dcf1c50119 100644 --- a/tasks/scripts/release.py +++ b/tasks/scripts/release.py @@ -341,14 +341,27 @@ def post_install system bin/"openshell-gateway", "generate-certs", "--output-dir", var/"openshell/tls", "--server-san", "host.openshell.internal" gateway_config = var/"openshell/gateway.toml" + gateway_config_contents = <<~TOML + [openshell] + version = 1 + + [openshell.gateway] + TOML + legacy_ipv6_gateway_config_contents = <<~TOML + [openshell] + version = 1 + + [openshell.gateway] + bind_address = "[::1]:{LOCAL_GATEWAY_PORT}" + TOML unless gateway_config.exist? - gateway_config.write <<~TOML - [openshell] - version = 1 - - [openshell.gateway] - bind_address = "[::1]:{LOCAL_GATEWAY_PORT}" - TOML + gateway_config.write gateway_config_contents + else + # Migrate only the exact config generated by the affected formula. Keep + # any user-edited config untouched. + if gateway_config.read == legacy_ipv6_gateway_config_contents + gateway_config.write gateway_config_contents + end end entitlements = var/"openshell/openshell-driver-vm.entitlements.plist" @@ -379,7 +392,7 @@ def caveats brew services restart openshell Register it with the OpenShell CLI: - openshell gateway add https://[::1]:{LOCAL_GATEWAY_PORT} --local --name openshell + openshell gateway add https://localhost:{LOCAL_GATEWAY_PORT} --local --name openshell EOS end diff --git a/tasks/scripts/test-install-sh.sh b/tasks/scripts/test-install-sh.sh index 88e08dfed1..5a20399b6a 100755 --- a/tasks/scripts/test-install-sh.sh +++ b/tasks/scripts/test-install-sh.sh @@ -100,8 +100,8 @@ assert_glibc_preflight_fails \ "OpenShell Linux packages require glibc >= 2.28; detected musl or unsupported libc." \ setup_ldd_musl -if [ "$(PLATFORM=darwin local_gateway_endpoint)" != "https://[::1]:17670" ]; then - echo "FAIL: macOS local gateway endpoint must use IPv6 loopback" >&2 +if [ "$(PLATFORM=darwin local_gateway_endpoint)" != "https://localhost:17670" ]; then + echo "FAIL: macOS local gateway endpoint must use a TLS-compatible loopback hostname" >&2 exit 1 fi