imagecache: record layer sizes at unpack and image last-use on cache hit - #650
Merged
Dmitry Berkovich (dberkov) merged 3 commits intoAug 1, 2026
Merged
Conversation
Data foundations for eviction (agent-substrate#463 Phase 2), inert on their own: - unpack counts the uncompressed tar stream and writes a per-layer 'size' file before the atomic rename, so sizing the pool never walks a layer tree; the write is best-effort (a missing size file is recovered by the lazy backfill); layers from older atelets are backfilled on first sizing (one walk, once ever), skipping unreadable dirs (atelet has no CAP_DAC_READ_SEARCH) and preserving the dir mtime - the cache-hit path touches the image record's mtime, persisting last-use for LRU ordering across atelet restarts; the touch happens under a new hitMu (shared side only — the eviction pass will be its exclusive holder) - Store.CacheSize sums recorded sizes, the accounting behind the future --image-cache-max-bytes Extracted from the phase2-gc-record-first prototype branch. This slice: go test -race green standalone; gofmt/vet clean; no behavior change for any current caller.
Comments now describe the mechanism's purpose (eviction, the GC root-set scan) without pointing at agent-substrate#463 phases, which would go stale once the work lands.
This was referenced Jul 31, 2026
…doc, corrupt-size-file test
Dmitry Berkovich (dberkov)
approved these changes
Aug 1, 2026
Dmitry Berkovich (dberkov)
merged commit Aug 1, 2026
51235e6
into
agent-substrate:main
11 checks passed
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Aug 1, 2026
…ep (#658) Third foundation slice of #463 Phase 2 (GC), independent of #650 and #656. The primitives are uncalled until the eviction engine PR; the only live behavior is the startup sweep, which only ever sees `.rm-*` dirs the engine will create. - **`retireLayer`**: evicts a layer with one atomic rename to a `.rm-*` name inside the layer singleflight; the slow `RemoveAll` is the caller's job, outside all locks. Existence/mtime pre-flight runs *outside* the flight (a retire must never block behind an in-progress download) and both checks are re-run inside it before the rename. Returns gone/vetoed/retired so the engine can distinguish "nothing there" from "must keep". - **Reuse interlock**: `ensureLayer` now refreshes the layer dir mtime inside the same flight, so a retirement and a reuse cannot interleave — either the retire wins (the pull re-unpacks) or the touch wins (the retire vetoes). - **Startup sweep**: `.rm-*` dirs (a crash between rename and removal) are reclaimed alongside the existing `.tmp-*` dirs, via `RemoveAllWritable` since layer trees legitimately contain read-only content. - **`isLayerDirName`** validates names read from disk or records. Divergence from the prototype: `retireLayer` reports no freed size — size crediting belongs to the engine's call sites and arrives with it. **Testing**: unit tests for the status contract, startup-sweep recovery and non-interference, and a retire-vs-pull race test; `go test -race` green.
Dmitry Berkovich (dberkov)
pushed a commit
that referenced
this pull request
Aug 1, 2026
…Spec (#656) Second foundation slice of #463 Phase 2 (GC), independent of #650 and inert on its own: nothing reads the new field and nothing deletes anything. - **`OverlaySpec.ImageDigest`** (optional, `omitempty`): the manifest digest the bundle's image ref resolved to, populated in `prepareOCIDirectory`. The GC root-set scan will use it to root the whole image record while the bundle exists. Compatible in both directions: ateom consumers never read the field (old binaries drop the unknown JSON key), and specs written by older atelets keep parsing with the field empty — so there is no deploy-ordering constraint. - **Atomic `WriteSpec`** (temp file + rename in the bundle dir): the root-set scan will read specs concurrently with bundle preparation, and a torn spec would under-report the layers an actor is using — an error in the dangerous direction (toward deleting in-use layers). With the rename, a reader sees the complete old spec or the complete new one, never a fragment. Landing early matters: every bundle written from now on carries the digest, shrinking the digestless-spec population the GC's compatibility fallback has to cover. **Testing**: round-trip + old-spec compat and no-temp-files-left unit tests; `go test -race` green on both affected packages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of #463 Phase 2 (GC): the two facts eviction will need, persisted in the
filesystem so they survive atelet restarts. Inert on its own — no behavior change for
any current caller, and nothing deletes anything.
sizefile before the atomic rename (best-effort; a missing file is healed by alazy backfill). Backfill covers layers unpacked by older atelets: one walk, once
ever, skipping unreadable dirs (atelet has no
CAP_DAC_READ_SEARCH) and preservingthe dir mtime.
timestamp — under a new
hitMu(shared side only; the eviction pass will be itsexclusive holder).
Store.CacheSize(): sums recorded sizes; the accounting behind the future--image-cache-max-bytes.On-disk changes are purely additive, so no cache-layout version bump and no
migration: an older atelet reads this cache dir fine (ignores
sizefiles), andthis atelet reads a pre-upgrade dir fine (backfill). Rollback-safe.
Testing: unit tests for size recording, backfill (incl. the unreadable-dir
contract and mtime preservation), hit-path touch, and
CacheSize;go test -racegreen standalone; the demo e2e suite passes unchanged on Kind against this build,
with
sizefiles and record-mtime updates verified on the node.