feat(plugin-oci): oci_image and oci_layer — container images without docker

[raphaelvigee]

Stacked on #159 (needs the oci_image → docker_build rename that landed there).

What

Two drivers that build a container image from target outputs — no Dockerfile, no BuildKit, no daemon, no execution.

oci_layer(name = "app", srcs = [":bin"], prefix = "/usr/bin")
oci_layer(name = "etc", srcs = [":conf"], prefix = "/etc")

oci_image(
    name = "img",
    base = ":alpine",                    # optional oci_pull(layout = True)
    layers = [":app", ":etc"],           # ordered, later wins
    platforms = ["linux/amd64", "linux/arm64"],
    entrypoint = ["/usr/bin/server"],
    env = {"PORT": "8080"},
)

Output is the same OCI layout docker_build emits, so oci_push, oci_load and bases consume it unchanged.

Why

Not "avoids a daemon". It is the only image rule whose cache key can cover what the build reads. docker_build's own docs concede the gap: the buildx version is unhashed, FROM is resolved by BuildKit from the network, RUN fetches whatever it fetches, secret values cannot be hashed. Here there is no host binary, no subprocess, no env var, no run-time network.

The docs scope that claim rather than overreach — a base is only as hermetic as the oci_pull behind it, and oci_pull keys on the ref string.

Second consequence: cross-arch is free. A layer is a file tree, so nothing executes for the target architecture. An arm64 Mac emits linux/amd64 and linux/arm64 in one run, no QEMU, same digests as CI.

Cache-key hazards fixed here, not later

Three ways two different images would have shared one cache entry:

1. hashin folds a sorted, unlabeled multiset of dep hashouts. layers = [":a", ":b"] and the swap reach it identically — as do base = ":a", layers = [":b"] and its mirror, and layers_by_platform with its values swapped. The def hash carries the ordered, normalized addresses. (docker_build's single dockerfile = ":t" role can safely omit its address — one occupant means any change moves that dep's hashout. An ordered list has no such property; the design's original "same rule as dockerfile" was the wrong generalization.)
2. inputs_result_meta folds every output group regardless of the |group selector, so [":bin|release"] and [":bin|debug"] were one key. The normalized address includes it.
3. Modes cannot be preserved. heph records one permission bit (walk/cached_walker.rs), the pack step normalizes to 0755/0644, and the sandbox mode then depends on the umask, on whether another target marked the dep read-only, and on a 1 MiB threshold choosing FUSE over unpack. Mode comes from the exec bit or an explicit hashed mode; setuid/setgid/sticky are rejected loudly.

Reproducibility

mtime 0, uid/gid 0, empty uname/gname, entries sorted by raw path bytes, headers built by hand (append_path copies mtime/uid/gid/mode off the filesystem; append_dir_all walks in read_dir order). Config JSON goes through serde_json, whose Map is a BTreeMap here — oci-spec's config uses HashMap for Labels and Rust's RandomState is seeded per process, so two runs on one machine would emit two byte orders. No created, no per-layer timestamps.

Layers are uncompressed. Spec-legal, and it keeps the layer digest out of the hands of whichever deflate backend cargo's feature resolution picks — a swap to zlib-ng would otherwise move every layer digest in every cache with no code change. It also stops gzipping bytes the remote cache is about to gzip again.

Silent failures made loud

Each with a test: empty layer (names what the srcs produced and which strip values would work), two entries mapping to one path, a .wh. whiteout name, an absolute symlink, a base with no matching platform, platforms unset. Symlinks are preserved, not followed — following one silently duplicates bytes or embeds an undeclared sandbox file.

Memory

Layer blobs travel as file paths, never Vec<u8>, so an image is not held in memory once per concurrent target. Blobs are written to a temp name and renamed: a blob's filename asserts a digest and nothing re-verifies it, so a Ctrl-C mid-write would otherwise leave a truncated file served from cache forever.

Tests

• 12 unit tests in plugin-oci (layer determinism, sort key, symlinks, modes, def-hash collisions, config merge matrix, canonical encoding)
• 9 engine e2e in crates/e2e/tests/oci_image.rs, none gated on docker — that is the feature
• a frozen manifest digest, which CI's three native targets turn into the cross-platform reproducibility guarantee. Two runs on one machine share a filesystem, a umask and an architecture — where the interesting divergences live.

Design

Four design-stage agent consults (product-vision, feature-quality, hermeticity, compatibility) shaped this. Three calls were the user's: rename oci_image → docker_build (in #159), platforms required with no default, and the DirPath sort fix as its own PR (#377).

Not in scope, documented: no RUN, no whiteouts/deletions, no hardlinks, gzip layers.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WqQZxsTqanJRWLWdPSchBo

[raphaelvigee]

Both follow-ups from the review are now in, so nothing is left deferred.

The docker_build slot-hashing bug is fixed in #159, its own base — context, context_by_platform and bases map a name to a target and only the names reached the key. All of these were one cache entry for two different images:

context = {"a": [":x"], "b": [":y"]}                 vs the swap
  -> SRC_A and SRC_B name each other's files
bases   = {"base": ":alpine", "tools": ":ubuntu"}    vs the swap
  -> `FROM base` resolves to the other image
context_by_platform amd64/arm64 with targets swapped
  -> the amd64 leg stages the arm64 binary
context = {"a": [":bin|release"]}                    vs [":bin|debug"]
  -> the selector reaches no hash at all

The def now carries the normalized TargetAddr (which includes |output) beside each name. DOCKER_BUILD_FORMAT_VERSION 4 → 5. dockerfile = ":target" still omits its address and that stays correct — a single-occupant role changes hashout whenever the target changes; generalizing that to an N-occupant mapping is what produced the bug. This PR is rebased on it.

The whole-image buffering is fixed here. Layout::read slurped every blob into a HashMap<String, Vec<u8>> and write_layout_dir opened with blobs.clone() on top — roughly 2× (base + own layers) per concurrent target. A Layout now records where each blob is:

• layout directory → the blobs are already files
• oci-archive → a tar, so a blob is a byte range (raw_file_position + size), no extraction
• only manifests, indexes and configs are ever read, and those are kilobytes

Both registry ends follow: push_layout uses push_blob_stream with a chunked reader instead of push_blob (which wants the layer as one Bytes), and pull_layout writes each blob to disk as it arrives via pull_blob_stream instead of accumulating the lot. Chunks go through std::fs, not tokio::fs — a cdylib's tokio is a separate runtime instance polled by host workers, so reaching for a reactor or a blocking pool aborts across the ABI seam.

Pulled blobs get the same temp+rename the layout writer already had: a blob's filename asserts a digest nothing re-verifies, so an interrupted pull must not leave a truncated file claiming to be the whole layer.

The buffering wrappers are deleted rather than kept as a convenience, so no future caller reintroduces the copy by reaching for the shorter name.

New tests: a structural one asserting a read layout hands back locations (Blob::FileRange from an archive, Blob::File from a directory) and that a blob located in one layout writes into another without being read; a chunking test for the push path; and a docker-format archive test — the last two previously had coverage only in the daemon-gated suite.