feat(plugin-oci): OCI/container image integration (build, push, load, pull)

[raphaelvigee]

What

A full OCI/container image integration for heph, as the plugin-oci crate. Four drivers, selected in a BUILD file via driver = "...".

It ships the way the go and gha plugins do — not compiled into the CLI, but as a per-os/arch cdylib published with a manifest a workspace opts into:

plugins:
  - url: https://…/heph-oci-plugin.json

oci_image — build (cacheable)

Builds a container image archive from a Dockerfile + build context via host docker buildx. Outputs the archive (<name>.tar) and a digest group.

target(
    name = "img", driver = "oci_image",
    context = {"": [":dockerfile"], "bin": ["//cmd/server:bin"]},
    dockerfile = "Dockerfile",
    build_args = {"VERSION": "1.2"},
    stage = "runtime",                    # multi-stage: buildx --target
    platforms = ["linux/amd64", "linux/arm64"],
    builder = "multi",                    # which buildx builder to build on
    bases = {"base": [":alpine"]},        # FROM base, from an oci_pull layout
    secrets = ["id=token,env=TOKEN"], ssh = ["default"],
    format = "oci",                       # "oci" (default) | "docker"
    cache_from = ["type=registry,ref=reg/img:cache"],
    cache_to = ["type=inline"],
)

The build context is the sandbox workspace root, so a context dep from any package is reachable and COPY paths are workspace-relative. Each context group is also exported as a SRC_<GROUP> build arg, so a Dockerfile need not hardcode the layout:

ARG SRC_BIN
COPY ${SRC_BIN} /usr/bin/server

oci_pull — base-image fetch (cacheable)

The image-world http_fetch: pulls a registry image into a cacheable archive so a shared base is pulled once. Warns on unpinned tags (pin @sha256: for reproducibility).

target(name = "alpine", driver = "oci_pull",
       ref = "docker.io/library/alpine@sha256:...",
       layout = True,          # OCI layout dir — the form `bases` consumes
       all_platforms = True)   # keep the whole index, for a multi-arch build

oci_push — push to registry (action, uncached)

skopeo copy --multi-arch all <transport>:<tar> docker://<ref> — daemonless, keeps a manifest list intact, skips blobs the registry already has. tool = "docker" does load + tag + push + rmi instead, for a docker-format archive.

target(name = "push", driver = "oci_push", image = ":img", ref = "reg.io/me/app:1.2")

oci_load — load into local daemon (action, uncached)

docker load -i for a docker archive; skopeo copy oci-archive:<tar> docker-daemon:<tag> for an OCI archive. A daemon tag holds one image, so a multi-arch archive is narrowed by platform (default linux/<host arch>).

target(name = "load", driver = "oci_load", image = ":img", tag = "app:dev")

Multi-stage and multi-arch

Multi-stage is stage → buildx --target. Multi-arch has one working route, and every step of it is exercised end to end against a real registry:

oci_pull(layout = True, all_platforms = True)   [skopeo]
  → oci_image(platforms = […], bases = {…})     [buildx, container builder, format = "oci"]
    → oci_push                                  [skopeo, --multi-arch all]
    → oci_load(platform = "linux/amd64")        [skopeo — a daemon tag holds one image]

A docker-CLI-only route (format = "docker", no skopeo anywhere) is fully supported but single-arch end to end; platforms with more than one entry and format = "docker" is rejected at parse.

Caching (the efficiency goal)

Layered, in order of impact:

1. heph input-hash cache (primary win) — oci_image inputs are the context files, Dockerfile, build args, stage, platforms, the resolved builder platform, base contexts, secrets and ssh specs. Unchanged inputs → cache hit → no rebuild; the archive is served from the local or remote cache. Image timestamp nondeterminism doesn't defeat this: heph keys on inputs and serves the identical cached archive to consumers.
2. BuildKit layer cache — cache_from / cache_to wire --cache-from/--cache-to (registry/inline). Deliberately excluded from the input hash (a build optimization, not part of the image's identity — changing them never busts the heph cache).
3. oci_pull base cache — base images cached (local + remote), pulled once.

What is in the key, and why

The remote cache key carries no OS or arch segment, so an unpinned platform is a wrong-artifact bug, not a miss:

• platforms empty → the builder is asked for its default platform at parse time (docker buildx inspect --bootstrap, memoized per builder) and the answer is hashed. Without it an arm64 laptop and an amd64 runner compute the same key for different images.
• builder is hashed — which builder runs decides the platforms, the BuildKit version and the layer cache. That is why BUILDX_BUILDER is stripped from the child environment: an ambient builder would change the image behind the key's back. What is hashed is the name, the same gap the docker version itself has.
• oci_pull always resolves to a concrete os/arch (or all), never "whatever the host is".

Known, deliberate exemptions are documented in the module header: unpinned FROM base images, whatever RUN fetches, secret/ssh values, the host docker/skopeo version, and .dockerignore.

Design

• oci_image / oci_pull are cacheable artifact drivers; oci_push / oci_load are CacheConfig::off() actions (external side effects, run every time).
• Push/load consume only the archive output group ("") of their image target — an explicit group selector is rejected rather than honoured.
• Every subprocess goes through hproc::proc_exec: cleared environment with an explicit passthrough allowlist, explicit cwd, output streamed to the user as it arrives, and the child SIGKILLed on cancellation (a detached buildx would otherwise keep writing into a sandbox the cleaner is deleting, and park runtime shutdown).
• Builder is host docker buildx / skopeo (host capability, like http_fetch's network). A hermetic toolchain can replace it later without changing targets.
• Modeled on the plugin-http ManagedDriver template; sandbox/staging/output-collection via ManagedDriverBridge.

Tests

86 unit tests (plugin-oci) — argv assembly, parse → def, hash semantics, and run() against fake binaries. Notably: cache refs do not affect the input hash while build args / secrets / ssh / builder / platform do; the child environment really is cleared.

5 engine e2e against a fake docker (crates/e2e/tests/oci.rs) — outputs are real artifacts a downstream target consumes, an unchanged context is a cache hit, a cross-package dep reaches the build as SRC_*.

10 e2e against the real docker and skopeo (crates/e2e/tests/oci_docker.rs) — every assertion is something only a real builder can answer: the archive is the format the driver claims, the digest is a real sha256, COPY ${SRC_BIN} resolves, stage narrows the build to one stage's DAG (proven by a sibling stage that cannot build), a multi-platform build indexes both architectures, builder works, and a full multi-arch push → pull → FROM base round trip against a throwaway registry:2. They skip rather than fail where the host cannot run them (macOS CI has no daemon); skopeo is on the dev shell, which is how CI gets it.

Three bugs the real-docker suite caught that the fake could not:

• bases never worked. A dep's unpack list names files, not the directories holding them, so the build context resolved to …/base.oci/oci-layout — a file. Every FROM <base> build failed.
• On a stock Docker Engine, no oci_image target can build at all. The plain docker driver has no file exporters, so --output type=oci,dest=… and type=docker,dest=… both fail regardless of format. BuildKit names the exporter but not the remedy; the driver now supplies it (create a docker-container builder and name it with builder =, or enable the containerd image store). Not checked at parse: buildx inspect cannot see the image store, so only the build itself avoids false negatives.
• oci_load failed outright on macOS for a multi-arch archive — skopeo matched the host's darwin/arm64 against a manifest list that has no such instance.

Toolchain requirements (runtime)

Host docker buildx (build) and skopeo (push/pull, and OCI load). The buildx builder must be able to write an image archive to a file: a docker-container builder, or a daemon with the containerd image store. Multi-platform builds need the same. A hermetic toolchain is a possible follow-up.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WqQZxsTqanJRWLWdPSchBo