ci(docker): bump buildcache to v6 (recurring short-read corruption) - #45
Merged
Conversation
Same short-read blob corruption pattern recurred on paperclip-agent base
build run 25983954502 (07:23:46Z):
buildx failed: failed to compute cache key:
short read: expected 32 bytes but got 0: unexpected EOF
Root cause is Ceph RGW zero-byte object corruption in the zot S3 backing
store (Phase A finding 2026-05-17 06:30Z): zot serves specific buildcache
layer blobs as `Content-Length: 0` even though their manifests declare
non-zero sizes. The canonical fix per `buildcache-v2 → v3 → v4 → v5` is
to bump the cache ref so buildkit abandons the corrupted cache and
rebuilds it from scratch.
This sweep covers the two refs that previously didn't get the v5 bump:
- docker.yml paperclip:buildcache-v5 → -v6
- docker-agent.yml paperclip-agent:buildcache → :buildcache-v6
- docker-mcp-gateway paperclip-mcp-gateway:buildcache → :buildcache-v6
Old corrupted refs (`-v5`, unversioned `:buildcache`) stay on zot; will be
swept in Phase D zot retirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
4 tasks
kkroo
added a commit
that referenced
this pull request
May 17, 2026
…32B blob as 0 bytes) (#47) `Docker (agent base)` has been failing for a week with `short read: expected 32 bytes but got 0: unexpected EOF` at FROM-layer resolution. The 17d handoff hypothesized buildx state corruption / attestation manifest issues and ruled out a Ceph RGW small-blob bug because two previously-tested blobs (756B + 263B) served correctly from zot. Today's check on the actually-failing blob shows the bug: curl -I https://registry.blockcast.net/v2/paperclip/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 HTTP/2 200 / content-length: 0 ← truncated to zero curl -I --resolve harbor.blockcast.net:443:69.25.95.125 \ https://harbor.blockcast.net/v2/paperclip/paperclip/blobs/sha256:4f4fb700... HTTP/2 200 / content-length: 32 ← correct Same Phase A zero-byte-blob pattern, different blob. zot's Ceph RGW backend silently zero-truncates some small blobs on read. Harbor's filesystem-backed registry-v2 storage serves the same blob correctly. Fix: swap the agent build's BASE_IMAGE pull + buildcache to harbor. Phase B (BLO-5717, #213 onprem-k8s + #42-#45 paperclip) already parallel-pushes server images to both registries, so the sha-tag exists in harbor when this workflow runs. The buildcache moves fresh (no buildcache-v6 has been published yet — the build has been failing all session). This is also strategic alignment with Phase D zot retirement (~3-7 days out per the harbor cutover plan). Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
kkroo
added a commit
that referenced
this pull request
May 17, 2026
…push target (#48) Harbor parallel-push (PR #43) has been live since 2026-05-17 07:38Z; the mcp-gateway build at 07:40Z verified end-to-end push to both registries successfully. Now ready for the next cleanup step in the zot retirement plan: drop the zot half of the parallel-push so builds land in Harbor only. This change unblocks the registry.blockcast.net DNS A-record flip from zot's IP (69.25.95.97) to Harbor's IP (69.25.95.125). After this PR merges and a build proves clean, any flat zot references (like buildcache pulls keyed on registry.blockcast.net/...) would resolve to Harbor and 404 on the flat path — so we have to retire them at the source first. Per workflow: - docker.yml (paperclip server image): * removed "Login to Blockcast registry" step * dropped the BLOCKCAST_REGISTRY_HOST entry from metadata-action's images: list * flipped cache-from/cache-to from registry.blockcast.net/paperclip to harbor.blockcast.net/paperclip/paperclip * dropped `vars.BLOCKCAST_REGISTRY_HOST != ''` from the deploy job gate (was used as a sentinel for "CI deploy enabled"; the dedicated `vars.PAPERCLIP_CI_DEPLOY == 'true'` check already covers that) - docker-agent.yml (paperclip-agent image): * removed "Login to Blockcast registry" step * dropped the BLOCKCAST_REGISTRY_HOST entry from images: * cache-from/cache-to were already on Harbor (PR #45) - docker-mcp-gateway.yml (paperclip-mcp-gateway image): * removed "Login to Blockcast registry" step * dropped the BLOCKCAST_REGISTRY_HOST entry from images: * flipped cache-from/cache-to to Harbor * updated stale comment that referenced BLOCKCAST_REGISTRY_PASSWORD Repo variable BLOCKCAST_REGISTRY_HOST and secret BLOCKCAST_REGISTRY_PASSWORD are now unreferenced; can be deleted from repo Settings as a follow-up. Buildcache impact: the first build per image after this lands will be cold (the v6 buildcache tags currently live in zot; Harbor's buildcache-v6 in each project is empty). Subsequent builds will be warm. Acceptable cost for cleanup. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bump cache refs in docker.yml, docker-agent.yml, docker-mcp-gateway.yml to v6 to abandon corrupted zot buildcache. Latest hit: agent run 25983954502 at 07:23:46Z. Root cause: Ceph RGW zero-byte objects in zot S3 backing store (Phase A audit this session).