Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,7 @@ Common findings:
- Docker daemon unavailable: start Docker Desktop or Docker Engine.
- Gateway process stopped: inspect exit status and logs.
- Sandbox image missing or pull denied: verify image reference and registry credentials.
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected.
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
- Sandbox never registers: check gateway logs and supervisor callback endpoint.
- Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`.
Expand All @@ -173,6 +174,7 @@ Common findings:
- Podman socket unavailable: start or expose the user socket.
- Rootless networking unavailable: inspect Podman network configuration.
- Sandbox image missing or pull denied: verify image reference and registry credentials.
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected.
- Supervisor cannot call back: check callback endpoint and gateway logs.

### Step 6: Check Kubernetes Helm Gateways
Expand Down
12 changes: 7 additions & 5 deletions .agents/skills/generate-sandbox-policy/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -473,15 +473,17 @@ filesystem_policy:
landlock:
compatibility: best_effort

process:
run_as_user: sandbox
run_as_group: sandbox

network_policies:
# <generated policies go here>
```

The `filesystem_policy`, `landlock`, and `process` sections above are sensible defaults. Tell the user these are defaults and may need adjustment for their environment. Gateway inference is configured separately through `openshell inference set/get`. The generated `network_policies` block is the primary output.
The `filesystem_policy` and `landlock` sections above are sensible defaults.
Process identity is omitted so the selected compute driver can choose it. For
Docker and Podman, each omitted identity field falls back to the image's OCI
`USER`. Tell the user these are defaults and may need adjustment for their
environment. Gateway inference is configured separately through `openshell
inference set/get`. The generated `network_policies` block is the primary
output.

If the user provides a file path, write to it. Otherwise, ask where to place it. A common convention is a project-local policy file (e.g., `sandbox-policy.yaml`) passed to `openshell sandbox create --policy <path>` or set via the `OPENSHELL_SANDBOX_POLICY` env var.

Expand Down
9 changes: 4 additions & 5 deletions .agents/skills/generate-sandbox-policy/examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -830,10 +830,6 @@ filesystem_policy:
landlock:
compatibility: best_effort
process:
run_as_user: sandbox
run_as_group: sandbox
network_policies:
github_readonly:
name: github_readonly
Expand All @@ -858,7 +854,10 @@ network_policies:
- { path: /usr/local/bin/claude }
```

The agent notes that `filesystem_policy`, `landlock`, and `process` are sensible defaults that may need adjustment, and that gateway inference is configured separately via `openshell inference set/get` rather than an `inference` policy block.
The agent notes that `filesystem_policy` and `landlock` are sensible defaults
that may need adjustment. Process identity is omitted so the compute driver can
select it. Gateway inference is configured separately via `openshell inference
set/get` rather than an `inference` policy block.

---

Expand Down
6 changes: 6 additions & 0 deletions .agents/skills/openshell-cli/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,12 @@ The `--from` flag accepts a Dockerfile path, a directory containing a Dockerfile

Local Dockerfile and directory builds require a local gateway because the CLI builds through the local Docker daemon. Use a registry image reference for remote gateways. Bare community names resolve under `ghcr.io/nvidia/openshell-community/sandboxes` unless `OPENSHELL_COMMUNITY_REGISTRY` overrides the prefix.

For Docker and Podman gateways, custom images should declare a non-root OCI
`USER`. Each explicit `process.run_as_user` or `process.run_as_group` policy
field wins independently; omitted fields fall back to the image declaration.
An image with no `USER` fails before readiness unless policy supplies both
fields.

### Forward ports

```bash
Expand Down
24 changes: 24 additions & 0 deletions architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,30 @@ Driver-controlled environment variables must override sandbox image or template
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
paths, and command metadata.

## Process Identity

The gateway preserves whether each policy process field was omitted. The active
driver then supplies one authoritative identity input to the supervisor:

- Docker and Podman inspect the final sandbox image, pin container creation to
its immutable image ID, and pass its raw OCI `Config.User`.
- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift
SCC-derived values.
- VM keeps its existing guest identity behavior.

For Docker and Podman, policy values take precedence independently. An omitted
`run_as_user` or `run_as_group` falls back to the corresponding identity from
the image. The supervisor resolves names from the image's `/etc/passwd` and
`/etc/group` before readiness, preserves declared name or numeric components,
and uses the same privilege-drop path for direct and SSH children. When a
declaration omits the group, the supervisor fills it with the user's numeric
primary GID. It does not rewrite the account files.

Sandbox creation fails before the workload becomes ready when a required image
identity is absent, malformed, unknown, ambiguous, or resolves to UID/GID 0.
The supervisor itself remains root so it can establish isolation before
starting unprivileged children.

Kubernetes can run the supervisor in the default combined topology or in a
sidecar topology. Combined mode keeps network and process supervision in the
agent container. Sidecar mode runs network enforcement, the proxy, and gateway
Expand Down
2 changes: 1 addition & 1 deletion architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ only when the set is already empty; any other outcome fails the spawn.
4. It starts the policy proxy and local SSH server.
5. It opens a supervisor session back to the gateway for connect, exec, file
sync, config polling, and log push.
6. It launches the agent command as the restricted sandbox user.
6. It launches the agent command as the resolved restricted identity.

## Isolation Layers

Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-core/src/sandbox_env.rs
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,17 @@ pub const SANDBOX_UID: &str = "OPENSHELL_SANDBOX_UID";
/// supervisor drops privileges to a group other than the UID's primary group.
pub const SANDBOX_GID: &str = "OPENSHELL_SANDBOX_GID";

/// Raw OCI `Config.User` declaration from the immutable image selected by a
/// local container driver.
///
/// Docker and Podman overwrite this value with the image declaration,
/// including an empty string when the image has no `USER`, and clear
/// [`SANDBOX_UID`] and [`SANDBOX_GID`]. Drivers with an authoritative numeric
/// identity overwrite this value with an empty string while supplying both
/// numeric fields. The supervisor resolves omitted policy identity fields from
/// OCI only for the former contract.
pub const OCI_IMAGE_USER: &str = "OPENSHELL_OCI_IMAGE_USER";

// The corporate upstream-proxy configuration deliberately has no reserved
// environment variables: it travels on the supervisor's argv
// (`--upstream-proxy` and friends), which a sandbox image cannot forge the
Expand Down
9 changes: 9 additions & 0 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,15 @@ The gateway runs as a host process. The Docker driver creates one container per
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
supervisor then creates the nested sandbox namespace for the agent process.

Before creating the container, the driver inspects the final sandbox image and
captures its immutable image ID and raw OCI `Config.User`. Container creation
uses that image ID, preventing a mutable tag from changing between inspection
and launch. The supervisor runs as root, resolves omitted policy identity fields
from the image declaration, and drops only agent children to the resulting
identity. Named OCI components remain names after validation; a missing group
is filled with the user's numeric primary GID. Explicit `process.run_as_user`
and `process.run_as_group` values take precedence independently.

Docker containers join an OpenShell-managed bridge network. The driver injects
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
names for reaching the gateway host. On Docker Desktop, Colima, Rancher
Expand Down
123 changes: 103 additions & 20 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,12 @@ struct DockerProvisioningFailure {
message: String,
}

#[derive(Debug, Clone, PartialEq, Eq)]
struct DockerImageMetadata {
id: String,
user: String,
}

#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
struct DockerResourceLimits {
nano_cpus: Option<i64>,
Expand Down Expand Up @@ -710,7 +716,8 @@ impl DockerComputeDriver {
DockerProvisioningFailure::new("ContainerCreateFailed", status.message())
})?;
let template = validated.template;
self.ensure_image_available(&sandbox.id, &template.image)
let image = self
.ensure_image_available(&sandbox.id, &template.image)
.await
.map_err(|status| {
DockerProvisioningFailure::new("ImagePullFailed", status.message())
Expand All @@ -735,11 +742,12 @@ impl DockerComputeDriver {
}
DockerProvisioningFailure::new("ContainerCreateFailed", status.message())
})?;
let create_body = build_container_create_body_with_gpu_devices(
let create_body = build_container_create_body_for_image(
sandbox,
&self.config,
&validated.driver_config,
gpu_devices.as_deref(),
&image,
)
.map_err(|status| {
if token_file_created {
Expand Down Expand Up @@ -1280,41 +1288,71 @@ impl DockerComputeDriver {
}))
}

async fn ensure_image_available(&self, sandbox_id: &str, image: &str) -> Result<(), Status> {
async fn ensure_image_available(
&self,
sandbox_id: &str,
image: &str,
) -> Result<DockerImageMetadata, Status> {
let policy = self.config.image_pull_policy.trim().to_ascii_lowercase();
match policy.as_str() {
let inspect = match policy.as_str() {
"" | "ifnotpresent" => {
if self.docker.inspect_image(image).await.is_ok() {
if let Ok(inspect) = self.docker.inspect_image(image).await {
self.publish_docker_progress(
sandbox_id,
"ImagePresent",
format!("Docker image \"{image}\" is already present"),
HashMap::from([("image_ref".to_string(), image.to_string())]),
);
return Ok(());
inspect
} else {
self.pull_image(sandbox_id, image).await?;
self.docker
.inspect_image(image)
.await
.map_err(|err| internal_status("inspect Docker image after pull", err))?
}
self.pull_image(sandbox_id, image).await
}
"always" => self.pull_image(sandbox_id, image).await,
"always" => {
self.pull_image(sandbox_id, image).await?;
self.docker
.inspect_image(image)
.await
.map_err(|err| internal_status("inspect Docker image after pull", err))?
}
"never" => match self.docker.inspect_image(image).await {
Ok(_) => {
Ok(inspect) => {
self.publish_docker_progress(
sandbox_id,
"ImagePresent",
format!("Docker image \"{image}\" is already present"),
HashMap::from([("image_ref".to_string(), image.to_string())]),
);
Ok(())
inspect
}
Err(err) if is_not_found_error(&err) => {
return Err(Status::failed_precondition(format!(
"docker image '{image}' is not present locally and image_pull_policy=Never"
)));
}
Err(err) if is_not_found_error(&err) => Err(Status::failed_precondition(format!(
"docker image '{image}' is not present locally and image_pull_policy=Never"
))),
Err(err) => Err(internal_status("inspect Docker image", err)),
Err(err) => return Err(internal_status("inspect Docker image", err)),
},
other => Err(Status::failed_precondition(format!(
"unsupported docker image_pull_policy '{other}'; expected Always, IfNotPresent, or Never",
))),
}
other => {
return Err(Status::failed_precondition(format!(
"unsupported docker image_pull_policy '{other}'; expected Always, IfNotPresent, or Never",
)));
}
};

let id = inspect.id.ok_or_else(|| {
Status::failed_precondition(format!(
"docker image '{image}' inspection did not return an immutable image ID"
))
})?;
let user = inspect
.config
.and_then(|config| config.user)
.unwrap_or_default();
Ok(DockerImageMetadata { id, user })
}

async fn pull_image(&self, sandbox_id: &str, image: &str) -> Result<(), Status> {
Expand Down Expand Up @@ -2132,7 +2170,16 @@ fn cleanup_sandbox_token_file_by_id(sandbox_id: &str, config: &DockerDriverRunti
}
}

#[cfg(test)]
fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) -> Vec<String> {
build_environment_for_oci_user(sandbox, config, "")
}

fn build_environment_for_oci_user(
sandbox: &DriverSandbox,
config: &DockerDriverRuntimeConfig,
oci_user: &str,
) -> Vec<String> {
let mut environment = HashMap::from([
("HOME".to_string(), "/root".to_string()),
("PATH".to_string(), SUPERVISOR_PATH.to_string()),
Expand Down Expand Up @@ -2204,6 +2251,18 @@ fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig

environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN);
environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN_FILE);
environment.insert(
openshell_core::sandbox_env::OCI_IMAGE_USER.to_string(),
oci_user.to_string(),
);
environment.insert(
openshell_core::sandbox_env::SANDBOX_UID.to_string(),
String::new(),
);
environment.insert(
openshell_core::sandbox_env::SANDBOX_GID.to_string(),
String::new(),
);

// Gateway-minted sandbox JWT. Keep the raw bearer out of container
// metadata; the supervisor reads it from this driver-owned bind mount.
Expand Down Expand Up @@ -2288,6 +2347,30 @@ fn build_container_create_body_with_gpu_devices(
config: &DockerDriverRuntimeConfig,
driver_config: &DockerSandboxDriverConfig,
gpu_device_ids: Option<&[String]>,
) -> Result<ContainerCreateBody, Status> {
let template = sandbox
.spec
.as_ref()
.and_then(|spec| spec.template.as_ref())
.ok_or_else(|| Status::invalid_argument("sandbox.spec.template is required"))?;
build_container_create_body_for_image(
sandbox,
config,
driver_config,
gpu_device_ids,
&DockerImageMetadata {
id: template.image.clone(),
user: String::new(),
},
)
}

fn build_container_create_body_for_image(
sandbox: &DriverSandbox,
config: &DockerDriverRuntimeConfig,
driver_config: &DockerSandboxDriverConfig,
gpu_device_ids: Option<&[String]>,
image: &DockerImageMetadata,
) -> Result<ContainerCreateBody, Status> {
let spec = sandbox
.spec
Expand Down Expand Up @@ -2328,9 +2411,9 @@ fn build_container_create_body_with_gpu_devices(
);

Ok(ContainerCreateBody {
image: Some(template.image.clone()),
image: Some(image.id.clone()),
user: Some("0".to_string()),
env: Some(build_environment(sandbox, config)),
env: Some(build_environment_for_oci_user(sandbox, config, &image.user)),
entrypoint: Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]),
// Clear the image CMD so Docker does not append inherited args to the
// supervisor entrypoint.
Expand Down
Loading
Loading