Large Image Streaming

Large-image streaming

The builder no longer reads the application, uncompressed tar stream, and compressed layer into memory. Its data path is:

source files -> deterministic PAX tar -> gzip -> content-addressed OCI blob
                    |                         |
                    +-- diff_id SHA-256       +-- descriptor SHA-256 + size

Both hashes are computed during the same pass. The final blob is renamed to its digest only after tar and gzip close successfully, and the complete layout is installed by an atomic rename.

Multi-terabyte inputs

PAX headers and signed 64-bit OCI descriptor sizes allow individual source files and aggregate layers beyond the historical USTAR limit. Copying uses a fixed-size buffer, so heap use does not grow with payload size. CI exercises a 64 MiB sparse input and fails if the build allocates 48 MiB or more; this is a scaling regression test, not a claim that a shared GitHub runner physically built a multi-terabyte image.

The copy path uses one 1 MiB cycling buffer per layer. The same allocation is reused sequentially for every source file instead of allocating in proportion to the file count or payload size. This improves large sequential reads while keeping memory predictable; it does not remove registry, disk, network, or runtime limits.

For very large inputs:

secure-oci build --compression=fast --output=oci-image ./service
# or
oci-builder -compression fast -binary ./service -output oci-image

best preserves the historical maximum-compression behavior. fast remains deterministic while reducing CPU time substantially.

Operational limits

Memory is no longer the payload-size limit, but the complete operation still requires:

• enough local disk for the compressed blob and temporary layout;
• enough time to read, hash, compress, upload, pull, and extract every byte;
• a registry and runtime whose per-blob and repository quotas admit the layer;
• enough node storage for Docker/Podman or KubeVirt to materialize it;
• an application format that does not require modifying a huge file in place.

A single multi-terabyte file remains a single OCI blob. OCI layers cannot concatenate fragments of one path: a later layer replaces that path. If the registry has a smaller blob limit, split application data into independent files or external immutable data volumes rather than pretending layer chunking can reconstruct one file.

Docker/Podman versus microVM

The same application can be selected at launch time:

secure-oci launch --isolation=container \
  --runtime=docker --network=bridge --publish=127.0.0.1:8080:8080/tcp \
  ghcr.io/acme/service@sha256:<digest>

secure-oci launch --isolation=microvm \
  --layout=oci-image --publish=127.0.0.1:8080:8080/tcp

The container path consumes a registry image through Docker or Podman. The microVM path consumes the verified local OCI layout and boots it with QEMU/KVM. The isolation choice does not change the application bytes.

Network contract

Container mode supports:

• isolated, bridge, host (explicit opt-in), and named runtime networks;
• repeated TCP/UDP publications with IPv4 or bracketed IPv6 host addresses;
• explicit DNS servers, hostname, and static host entries.

Native microVM mode accepts repeated TCP/UDP QEMU user-network forwards. KubeVirt manifests expose the corresponding guest ports on the masquerade interface. Kubernetes Services, NetworkPolicies, ingress, load balancers, and cluster DNS remain cluster resources and should be managed by the platform, not silently created by this CLI.

Host networking requires --allow-host-network. Port publication is rejected with network=none. Every address, port, protocol, hostname, and host entry is validated before an argument reaches the selected runtime.