fix(e2e): align container engine selection across helpers

[elezar]

Agent Diagnostic

The container engine selection behavior differs between the Rust e2e harness and the shell build helper:

• e2e/rust/src/harness/container.rs reads OPENSHELL_E2E_CONTAINER_ENGINE, then derives Podman only when OPENSHELL_E2E_DRIVER=podman, otherwise it defaults to Docker.
• tasks/scripts/container-engine.sh reads CONTAINER_ENGINE, then auto-detects Podman first when installed, then falls back to Docker.

This means a host with both Docker and Podman installed can run Rust e2e helper code with Docker while shell build scripts select Podman by default.

Description

Actual behavior: OpenShell has multiple container engine selection paths with different environment variables and default behavior. Rust e2e helpers and shell build helpers can select different engines on the same host.

Expected behavior: Container engine selection should have one documented precedence order across Rust e2e helpers and shell scripts. The same environment should select the same engine unless a caller explicitly overrides it.

Reproduction Steps

1. Use a host with both docker and podman installed.
2. Leave OPENSHELL_E2E_CONTAINER_ENGINE, OPENSHELL_E2E_DRIVER, and CONTAINER_ENGINE unset.
3. Observe that Rust e2e helper logic defaults to Docker.
4. Observe that tasks/scripts/container-engine.sh auto-detects Podman first.

Environment

• OS: Linux host with Docker and Podman installed
• Docker: any installed version
• Podman: any installed version
• OpenShell: current main

Logs

Rust helper:
  OPENSHELL_E2E_CONTAINER_ENGINE -> OPENSHELL_E2E_DRIVER=podman -> docker fallback

Shell helper:
  CONTAINER_ENGINE -> podman if installed -> docker fallback

Suggested Fix

Define and document a single precedence order, then update both implementations. One possible order:

1. explicit OPENSHELL_E2E_CONTAINER_ENGINE
2. explicit CONTAINER_ENGINE
3. driver-derived default from OPENSHELL_E2E_DRIVER
4. auto-detect an available engine
5. fail clearly if neither Docker nor Podman is available

Update:

• e2e/rust/src/harness/container.rs
• tasks/scripts/container-engine.sh
• relevant e2e/build documentation

Related Work

This came up while designing #1476. The GPU workload image build task should use the existing shell helper for now and avoid introducing another container-engine selector.

[TaylorMutch]

I've run into this as well. Thank you for the issue!

[jganoff]

I just ran into this as well, and also with the Helm / Skaffold dev setup scripts. Including some Claude-gathered context below for reference, since it may be helpful to fix/align in 1 pass.

---

This also bites outside the e2e harness, in the helm/Skaffold local dev path. Sharing the repro in case it tightens the fix scope.

Repro

macOS with Docker Desktop running and Podman installed but its machine not running:

mise run helm:k3s:create
mise run helm:skaffold:dev

Output:

[container-engine] auto-detected: podman (override with CONTAINER_ENGINE=docker|podman)
Staging prebuilt Rust binaries for Docker target 'gateway'...
unsupported architecture: OS: darwin/arm64

Workaround: CONTAINER_ENGINE=docker mise run helm:skaffold:dev.

Why this case adds something to the precedence design

The order proposed in the issue body would fix the inconsistency between the Rust e2e harness and container-engine.sh. The helm/Skaffold path has a stronger constraint worth encoding too:

The engine used to build an image must match the engine the destination cluster runs on.

Skaffold names it itself in the build log: Found [k3d-openshell-dev-...] context, using local docker daemon. k3d ran on Docker; a podman-built image is not visible to that daemon. Had podman info returned a valid arch, the build would have succeeded and deploy would have failed with ImagePullBackOff instead. Today's loud build failure is the milder symptom of the same root issue.

I'd add a step to the precedence list:

4. derive from kubectl config current-context when a kube context is the build target (e.g. k3d-* → docker, kind-on-podman → podman). Below explicit env overrides, above bare host auto-detect.

Adjacent bug, separately filable

ce_info_arch (tasks/scripts/container-engine.sh:176-182) suppresses podman info errors with 2>/dev/null and returns whatever lands on stdout. With a non-running podman machine it returned OS: darwin/arm64, which then flowed through normalize_arch → PREBUILT_ARCH → target_triple and finally surfaced as unsupported architecture: OS: darwin/arm64 from stage-prebuilt-binaries.sh four scripts later. Validating the arch string inside ce_info_arch would turn this into a clear "is your container engine running?" error.