Make published container images reproducible

[jetersen]

Publishing the same application twice produces two different image digests. Nothing about the build changed, but the registry cannot deduplicate the blobs and anything watching it treats the rebuild as a new artifact. This came up in a GitOps pipeline, where retrying a failed publish of one commit created a second, spurious release of an unchanged application.

Image creation is a function of the current time in four places:

• every layer tar entry is stamped with the moment it happened to be written. PaxTarEntry defaults its modification time to DateTime.UtcNow, and the file's own timestamp is never read, so each entry samples the clock separately;
• the image config samples DateTime.UtcNow twice, once for created and once for the generated history entries;
• the generated org.opencontainers.image.created label uses UtcNow in the targets file;
• every layer tar embeds the process id, because TarWriter names each pax extended header ./PaxHeaders.<process id>/..

The first and last dominate: either alone changes the layer digest even when the content is byte-identical.

What this changes

The three timestamp sites use SOURCE_DATE_EPOCH when it is set, which is the cross-ecosystem convention for exactly this problem. When it is unset the behavior is unchanged, so this part is opt-in. Following the specification, a value that cannot be interpreted is ignored rather than failing the build.

The pax header name is replaced with a constant. The name is not meaningful to an extractor: the path an extended header applies to is carried in its path record, not in its entry name, and the header always applies to the entry that immediately follows it. POSIX suggests the process id so that two concurrent extractions cannot collide over a temporary name, which does not apply to an archive being written to a stream. This is done with a small write-through stream between the tar writer and the stream that hashes the layer, so the digest is computed over the normalized bytes. It reassembles 512 byte blocks itself, since a caller can write across block boundaries, and it tracks each header's declared size so that file content which happens to look like a header is never rewritten.

Directory enumeration order is filesystem-defined, so entries are also sorted by their path in the container to keep the tar stream stable across machines.

Note that the pax change affects every publish, not only opted-in builds. Digests will shift once for everyone. That seemed right given it is fixing a defect rather than adding a behavior, but I am happy to gate it if you would prefer.

Verification

Published the same project twice to a registry and compared the manifest digests.

Before, the two publishes differed. Investigating showed only 13 bytes of the layer differed, and all of them followed from the pax entry name, the rest being the header checksums it shifts.

After:

scenario	digest
republish, no changes	sha256:2e726658… → sha256:2e726658… identical
one line of source changed	sha256:6354b493… differs
base image 10.0-noble	sha256:006bdce… differs
base image 10.0-alpine	sha256:b0c7746… differs

So the digest is still a function of the inputs; it is reproducible, not frozen. Also confirmed that with SOURCE_DATE_EPOCH unset the created label is still the current time.

Tests

Added coverage for the timestamp helper (including malformed values, culture independence and the unset case), for the normalizing stream (chunked writes, archive still reads back correctly, file content that looks like a header is untouched), and for layer reproducibility, including that layers built from different content still differ.

I checked these fail without the fix: reverting the entry timestamp change fails two of the three layer tests, while the "different content still differs" test keeps passing.

Addresses dotnet/sdk-container-builds#34 (determinism for layers) and dotnet/sdk-container-builds#585 (SOURCE_DATE_EPOCH support), which are called out as prerequisites in #54038.

Also covers most of #52256, which asks for control over the creation timestamps for the same reproducibility reason. Of the options suggested there, this implements the SOURCE_DATE_EPOCH convention, which subsumes "set it to the commit timestamp" (SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)) without the SDK needing to shell out to git.

One thing that issue reports is worth separating out. Setting the label directly does get overwritten:

<ContainerLabel Include="org.opencontainers.image.created" Value="2025-12-20T00:33:31.4004695Z" />

I confirmed this on a current SDK; the value is silently replaced with the publish time. But that is because the generated label is added to the same item group and wins, and there is already a supported way to yield to the user:

-p:ContainerGenerateLabelsImageCreated=false

With that set, I confirmed the user-specified value survives to the image config. So that half is arguably a discoverability problem rather than a missing feature, and I have left it alone here rather than changing item precedence in this PR.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

This PR makes PublishContainer outputs reproducible across repeated publishes of identical inputs by removing time/process-id nondeterminism from layer tar generation, image config creation timestamps, and the org.opencontainers.image.created label. This supports registry-side deduplication and enables downstream workflows (e.g., GitOps retry/promotion) to treat identical rebuilds as the same artifact.

Changes:

• Honor SOURCE_DATE_EPOCH (when set) for layer entry mtimes, image config created/history timestamps, and the generated OCI created label.
• Normalize pax extended header entry names to remove process-id variance and sort directory enumeration by container path to stabilize tar entry order.
• Add unit tests covering SOURCE_DATE_EPOCH parsing, pax header name normalization correctness, and layer digest reproducibility.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/Microsoft.NET.Build.Containers.UnitTests/SourceDateEpochTests.cs	Adds unit coverage for SOURCE_DATE_EPOCH parsing behavior (valid/invalid/unset/culture).
test/Microsoft.NET.Build.Containers.UnitTests/PaxHeaderNameNormalizingStreamTests.cs	Adds tests ensuring pax header renaming is deterministic and does not corrupt tar archives.
test/Microsoft.NET.Build.Containers.UnitTests/LayerReproducibilityTests.cs	Adds tests asserting identical inputs yield identical layer digests and entries use the pinned timestamp.
src/Containers/packaging/build/Microsoft.NET.Build.Containers.targets	Updates generated org.opencontainers.image.created label to use SOURCE_DATE_EPOCH when available.
src/Containers/Microsoft.NET.Build.Containers/SourceDateEpoch.cs	Introduces a helper to compute a stable UTC timestamp from SOURCE_DATE_EPOCH with fallback behavior.
src/Containers/Microsoft.NET.Build.Containers/PaxHeaderNameNormalizingStream.cs	Introduces a stream filter to rewrite pax header names deterministically while preserving tar validity.
src/Containers/Microsoft.NET.Build.Containers/Layer.cs	Makes layer tar generation deterministic (pax name normalization, stable mtimes, sorted entries).
src/Containers/Microsoft.NET.Build.Containers/ImageConfig.cs	Makes config created and history timestamps deterministic and consistent within the blob.

Suppressed comments (2)

test/Microsoft.NET.Build.Containers.UnitTests/LayerReproducibilityTests.cs:79

• This test sets SOURCE_DATE_EPOCH and then clears it to null in the finally block, which can clobber a pre-existing value in the test process. Capture and restore the original environment variable value instead.

        finally
        {
            Environment.SetEnvironmentVariable("SOURCE_DATE_EPOCH", null);
        }

test/Microsoft.NET.Build.Containers.UnitTests/LayerReproducibilityTests.cs:97

• This test sets SOURCE_DATE_EPOCH and then clears it to null in the finally block, which can clobber a pre-existing value in the test process. Capture and restore the original environment variable value instead.

        finally
        {
            Environment.SetEnvironmentVariable("SOURCE_DATE_EPOCH", null);
        }

[jetersen]

Thanks, one of these was a real bug.

Out-of-range SOURCE_DATE_EPOCH in the targets file: correct, now fixed. I ran each input through MSBuild to check:

SOURCE_DATE_EPOCH	before	after
1636374896	2021-11-08T12:34:56Z	2021-11-08T12:34:56Z
1,636,374,896	ignored, falls back	ignored, falls back
1636374896	ignored, falls back	ignored, falls back
abc / -5 / empty	ignored, falls back	ignored, falls back
99999999999999999999	error MSB4186	ignored, falls back

So the comma and whitespace cases were already safe: the regex rejects them and evaluation falls back to UtcNow, because the property is only expanded into FromUnixTimeSeconds after that guard clears it. But the out-of-range case did break the build exactly as you describe, since DateTimeOffset.FromUnixTimeSeconds throws. The C# helper already caught this; the targets file did not.

Fixed by bounding the digit count, which keeps every value inside the supported range:

'^[0-9]{1,11}$'

I also quoted the property in the IsMatch call as you suggested. It was not reachable as a break given the guard ordering, but quoting is correct and costs nothing.

Test environment variable restore: applied. Agreed, and it is cheap to be correct. The three sites in LayerReproducibilityTests now capture the previous value and restore it instead of clearing to null. SourceDateEpochTests already did this.

Full suite after both changes: 320 total, 314 passed, 6 skipped, 0 failed.

[jetersen]

Not sure if this deserves a fix in TarWriter implementation, but the quickest path for Build Containers was this stream fix. Opened dotnet/runtime#132049