Cross-run publish skip via registry as oracle

[BeyondEvil]

Cross-run publish skip via registry as oracle

Problem

When dotnet publish -t:PublishContainer runs on a fresh CI runner with a clean checkout, there is no local filesystem state to reason about. Every run rebuilds and re-pushes the image, even when the produced image would be byte-for-byte identical to what the destination registry already holds.

The existing incrementality work (#438, #590) is aimed at MSBuild's intra-machine incrementality, comparing timestamps in obj/ across successive local invocations, which is valuable for developer inner-loop scenarios but does nothing for CI. Fresh runners have no prior obj/ to inspect; the registry itself is the only persistent record of what has already been built.

BuildKit addresses this for Dockerfile builds via --cache-from=type=registry / --cache-to=type=registry, pushing and pulling cached layers through the registry itself. PublishContainer has no equivalent mechanism today, so .NET teams publishing multiple services to the same registry end up paying the full build + push cost on every commit, even when the resulting image digest would be unchanged.

Proposal

Add a <ContainerSkipPushIfAlreadyPresent> MSBuild property (default false). When set to true:

1. ImageBuilder.Build() already computes BuiltImage.ManifestDigest locally before any push begins, this work stays in place.
2. Before the layer / config / manifest PUT loop in Registry.PushAsync, HEAD the destination for /v2/{repository}/manifests/{ManifestDigest}.
3. If the manifest already exists at the destination (200):
   • Skip all layer uploads.
   • Skip the config upload.
   • Still PUT the requested ContainerImageTags against the destination — the registry will associate the new tags with the existing manifest digest. No blob data is transferred.
4. If the manifest does not exist (404): proceed with the current push flow.

This mirrors the spirit of the existing blob ExistsAsync check (Registry.cs:588), lifted to the manifest level. It is a pure optimization — when the digest differs, we fall through to the existing code path, preserving current semantics.

What this does not cover, and why that's fine

• Local daemon path (DestinationImageReferenceKind.LocalRegistry): not addressed. The feature targets remote registries, which is where CI pushes normally land.
• Intra-machine / developer inner loop: already covered by the ongoing #438 / #590 work. The two are complementary, not overlapping.
• Multi-architecture image indexes: initial proposal covers single-manifest publishes. Extending the check to PushManifestListAsync (the image-index digest) is a clean follow-up, same shape of change.

Dependency on layer determinism

The feature is most valuable once the produced layer digest is stable across independent builds of the same source. #34 (Determinism for layers) and #585 (SOURCE_DATE_EPOCH support) are prerequisites for the common case, otherwise the manifest digest varies run to run even when source is unchanged, and the skip check never fires.

Happy to pick up #34 / #585 as part of this effort: sort the FileSystemEnumerable output in Layer.FromDirectory, normalize tar entry mtime / uid / gid, and honor SOURCE_DATE_EPOCH where present. Those changes are self-contained and benefit the ecosystem beyond this specific feature.

Fallback path for non-deterministic builds

For callers who cannot rely on determinism yet but still want cross-run dedup today, a secondary property <ContainerCacheKey> would let the caller supply an opaque content hash computed from their own inputs (project sources, lockfiles, base image digest, etc.). The SDK would:

1. HEAD the destination for /v2/{repo}/manifests/content-{key}.
2. If present: PUT all ContainerImageTags against that digest and skip the build entirely - no publish, no tar, no push.
3. If absent: build + push as today, additionally tagging the result with content-{key}.

This is effectively "let the registry be your Nix store." Works without layer determinism because the cache key is derived from inputs, not from produced bits. Gates on reproducibility fully drop once #34 / #585 land.

API surface

• Add ExistsAsync(string repositoryName, string reference, CancellationToken cancellationToken) to IManifestOperations.
• Implementation in DefaultManifestOperations: HEAD /v2/{repo}/manifests/{reference} with the manifest Accept headers, return true on 200, false on 404. Other status codes fall back to false (treat as "doesn't exist") and let the downstream push attempt surface the real error, mirroring existing blob-exists behavior.
• Thread a bool skipIfManifestExists parameter through Registry.PushAsync, ContainerBuilder.ContainerizeAsync, and the CreateNewImage MSBuild task.
• Emit a telemetry/log event on skip so CI logs clearly indicate "image already present, tags updated without rebuild."

Tests

• Unit: DefaultManifestOperations.ExistsAsync returns true/false for 200/404, false for auth failures, propagates on server errors.
• Unit: Registry.PushAsync with skipIfManifestExists=true and a mocked registry reporting manifest present - assert zero layer/config PUT calls and one tag PUT per requested tag.
• Integration: publish once against a local test registry; publish a second time with the property on; assert blob count in the registry is unchanged and tags were updated.

Open questions

1. Property naming. ContainerSkipPushIfAlreadyPresent is descriptive but verbose. Alternatives: ContainerCheckManifestBeforePush, ContainerIdempotentPublish. Open to preferences.
2. Default value. Default off avoids behavior change for existing callers. Could default on once layer determinism is landed and the skip semantics are battle-tested.
3. Authorization. HEAD on manifests requires pull permission on the repository, which a pushing credential typically already has (push implies pull). Worth confirming this assumption against the OCI distribution spec.

Relation to #633(Layered Image Builds)

Orthogonal. Layered builds affect what gets pushed; this proposal affects whether to push at all. They compose cleanly: a layered image whose manifest digest has been seen before still benefits from the skip check.

Happy to take this on and pair with maintainers on scoping. If the direction is agreeable, I can start with #34 / #585 as stepping stones and land this as a follow-up once layer output is deterministic.