Add stream-based copyIn/copyOut for tar stream cp support - #812
Open
yash-mandaviya wants to merge 4 commits into
Open
Add stream-based copyIn/copyOut for tar stream cp support#812yash-mandaviya wants to merge 4 commits into
yash-mandaviya wants to merge 4 commits into
Conversation
Adds file-handle/stream-based copy entrypoints so callers can transfer a tar stream directly to/from a container's filesystem without staging the contents on the host, enabling true `docker cp -` / `podman cp -` parity in downstream CLIs (apple/container#1908). - LinuxContainer.copyIn(archive:to:) splices a tar stream from a file handle straight to the guest over vsock; the guest extracts it in place, honoring the ownership/mode/symlink metadata in the tar headers. - LinuxContainer.copyOut(from:to:) streams the guest-produced tar archive to a file handle; it always archives, even a single regular file, to match `docker cp CONTAINER:/path -`. - ArchiveReader.init(fileHandle:) auto-detects archive format and compression filter, so an externally supplied tar stream (uncompressed or gzip/bzip2/xz) is accepted, not only the internal pax+gzip form. - vminitd copy handlers use the auto-detecting reader for copyIn and honor CopyRequest.is_archive to force single-file archiving on copyOut. No host-side temp staging, so host permission and path-length limits no longer constrain the transfer. Existing path-based copyIn/copyOut behavior is unchanged (the new is_archive-on-copyOut path is opt-in). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The stream copyOut path reused archiveDirectory for directories, which emits a leading "./" and names entries relative to the directory itself, and always compressed with gzip. Docker and podman name entries relative to the source's parent, so the source's own basename is the top level entry, and emit an uncompressed tar. The difference is visible in a round trip: with contents-relative naming, `cp ctr:/dir - | cp - other:/dest` scatters dir's children into /dest instead of producing /dest/dir. The gzip framing also breaks consumers that read the stream with a plain tar reader rather than shelling out to tar. Use the parent-relative form for both files and directories whenever request.isArchive marks the stream path, with no compression filter. The internal directory copyOut, where isArchive is derived from the path being a directory, keeps shipping pax+gzip of the directory's contents, which is what the host side extracts into the destination directory.
yash-mandaviya
added a commit
to yash-mandaviya/container
that referenced
this pull request
Aug 1, 2026
The first pass wrapped the existing path based cp: it shelled out to /usr/bin/tar, unpacked the stdin archive into a host temp directory, then copied the extracted files in, and did the reverse on the way out. That gave up the two things tar streaming is for. Ownership and mode came from whatever the host filesystem allowed rather than from the tar headers, and every entry had to fit the host's path length limits under a temp directory even when its path was valid inside the archive. Hand the descriptor down instead. `cp -` now passes the CLI's stdin or stdout through both XPC hops on a new archiveFd key, using the same file handle passing that dial and logs already rely on, to LinuxContainer's stream based copyIn(archive:)/copyOut(to:). The bytes go straight to the guest over vsock and the guest extracts them as root, so the ownership, mode and symlink targets in the headers are applied verbatim and nothing is unpacked on the host. Path traversal is rejected during extraction, where the archive is actually read, rather than by pre-scanning entry names on the host. Each hop closes the descriptor it owns and duplicates for the outgoing message. Setting a file handle on an XPC message closes the descriptor it is given, and the container lookups on the receiving side can throw before the handle is ever forwarded, which otherwise leaked an fd in the apiserver on every copy to a stopped container. Behavior now follows docker and podman: output is uncompressed tar named relative to the source's parent so the source's basename is the top level entry, for files as well as directories, and input may be uncompressed or gzip, bzip2 or xz compressed. The integration tests assert metadata rather than content: a stream carrying uid, gid and modes the host could not reproduce unprivileged, a traversal entry that must not escape, entry naming on the way out, and a round trip. Requires the stream based copy API from apple/containerization#812; the containerization pin still needs bumping once that lands.
yash-mandaviya
marked this pull request as ready for review
August 2, 2026 00:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Downstream,
apple/containerwantscontainer cpto supportdocker cp -/podman cp -style tar streaming (apple/container#1908). The value of that feature is exact control over ownership and mode: the tar headers, not host filesystem metadata, define what lands in the container — injecting certs and config as a dynamic image layer.LinuxContainer.copyIn/copyOutare path-based. A downstream CLI can only implement streaming by unpacking to a host temp dir and re-copying, which loses uid/gid/mode fidelity (a non-root host cannot hold arbitrary ownership, and the re-copy reads host metadata) and hits host path-length limits. There is no entrypoint that accepts an already-formed tar stream.The guest and
ContainerizationArchivealready do metadata-preserving tar streaming over vsock. This exposes that directly, with no host staging.Changes
LinuxContainer.copyIn(archive:to:)— splices a tar stream from aFileHandlestraight to the guest over vsock (is_archive=true); the guest extracts in place honoring tar-header ownership, mode and symlinks. No host temp files.LinuxContainer.copyOut(from:to:)— streams the guest-produced tar to aFileHandle, always archiving, even a single regular file, matchingdocker cp CONTAINER:/path -.ArchiveReader.init(fileHandle:)— auto-detects format and compression filter, so an externally supplied stream (uncompressed, gzip, bzip2, xz) is accepted, not only the internal pax+gzip form. Mirrors the existinginit(file:)auto-detect for a stream source.vminitdcopy handlers —handleCopyInuses the auto-detecting reader;handleCopyOuthonorsCopyRequest.is_archive.archiveDirectory, which emits a leading./with contents-relative names, and always gzipped. The old behaviour broke round trips (cp ctr:/dir - | cp - other:/destscattereddir's children into/dest) and the gzip framing broke consumers using a plain tar reader.No proto change:
CopyRequest.is_archivealready exists; this repurposes it for the copyOut direction. Path-basedcopyIn/copyOutbehaviour is unchanged — the forced-archive path is opt-in, the internal directory copyOut still ships pax+gzip of the directory's contents, and the auto-detect reader is a strict superset of the previous hardcoded one.Testing
ContainerizationandContainerizationArchivehost targets build;vminitdbuilds foraarch64-swift-linux-musl.ArchiveReaderTests: 25/25, including new cases provingArchiveReader(fileHandle:)reads both uncompressed and gzip tar streams from a file handle.TestCLICopyCommand42/42 against a locally builtvminitdand initfs, with tests asserting ownership, mode, symlink targets, traversal rejection, entry naming and a round trip.Known gaps
The guest extractor applies mode and uid/gid but not mtime or xattrs, does not
lchownsymlinks, and silently skips hardlinks, devices and FIFOs (the copy still succeeds). These predate this PR and are outside the scope in apple/container#1908, but they're the remaining distance to 1:1 docker parity.setFileAttributesis shared with OCI layer extraction, so changing it is deliberately not a drive-by here.