Skip to content

feat(cache): add tar.gz pack/unpack helpers for remote-cache artifacts - #12

Merged
Exelord merged 1 commit into
mainfrom
claude/cache-archive
May 11, 2026
Merged

feat(cache): add tar.gz pack/unpack helpers for remote-cache artifacts#12
Exelord merged 1 commit into
mainfrom
claude/cache-archive

Conversation

@Exelord

@Exelord Exelord commented May 11, 2026

Copy link
Copy Markdown
Member

Summary

Pure tar+gzip helpers used by the upcoming LayeredCache to bridge the v10 on-disk layout and the Turbo /v8/artifacts wire body.

packArchive(stageDir): Promise<Uint8Array>        // tar -cz, returns bytes
unpackArchive(buf, destDir): Promise<void>        // tar -xz from stdin, mkdir -p destDir
packAndDiscard(stageDir): Promise<Uint8Array>     // pack then rm -rf the stage
tarPath(...segs): string                          // POSIX-safe path join
uniqueStageDir(parent, prefix): string            // pid+timestamp suffix

Implementation note: shells out to system tar via Bun.spawn — streaming stdin/stdout, no temp files for the archive bytes themselves. tar ships with every supported platform (GNU/BSD on Linux/Mac, MS-bundled on Windows 10+), so this avoids adding a JS tar dep.

What this enables (PR #11b)

The caller will build a stage dir matching the design doc layout:

stage/
  meta.json
  outputs/
    <project-relative paths>

…and packArchive(stage) becomes the PUT body. On GET, unpackArchive(body, stage) reproduces the same tree, which the LayeredCache then materializes into the v10 local layout (DB row + <hash>/<rel paths> + logs/<hash>.{stdout,stderr}).

Test plan

  • bun run format:check — clean
  • bun run lint — clean
  • bun test src/ — 209 / 209 (was 197; +12)

New tests (src/cache-archive.test.ts)

  • Round-trip: stage with meta.json + outputs/ survives pack→unpack with identical content.
  • Binary preservation: 256-byte all-values blob compares byte-for-byte.
  • Empty stage dirs pack and unpack cleanly.
  • Deep tree (6 levels) survives the round-trip.
  • unpackArchive creates destDir if missing.
  • packArchive rejects on missing source.
  • unpackArchive rejects corrupt tarballs.
  • Accepts ArrayBuffer and Uint8Array inputs.
  • tarPath: backslash normalization, double-slash collapse.

https://claude.ai/code/session_016HXj6HW6bxSn8EYuKcxTD9


Generated by Claude Code

Pure helpers that bridge the v10 on-disk layout and the Turbo
/v8/artifacts wire body. Shells out to system tar (GNU on Linux,
BSD on macOS, MS bundled on Windows 10+) via Bun.spawn — streaming
stdin/stdout, no temp files.

Public surface
- packArchive(stageDir) → Uint8Array: tar+gzip everything under stageDir.
- unpackArchive(buf, destDir): extract a tar.gz into destDir, mkdir -p first.
- packAndDiscard(stageDir): pack then rm -rf the stage. Convenience.
- tarPath(...segs): POSIX-safe join (normalizes backslashes, collapses //).
- uniqueStageDir(parent, prefix): pid+timestamp-suffixed path for races.

The caller (LayeredCache, next PR) stages a tree shaped per the
remote-cache design doc:
  meta.json
  outputs/<project-relative paths>

…then packs it for the wire, or unpacks the wire body back into a
stage dir before materializing into the local cache.

Tests: 12 new (209 total, up from 197). Cover round-trip, binary
preservation, empty dirs, deep trees, ArrayBuffer input, error paths
(missing source dir, corrupt tarball), tarPath edge cases.
@Exelord
Exelord merged commit dff8557 into main May 11, 2026
1 check failed
Exelord added a commit that referenced this pull request Aug 24, 2026
The fourth output mode (Turbo parity, comparison gap #7): one line per
task — outcome word, task id, cache key — with zero log output. The
run's audit trail of which key each task resolved to. discardsOutput
covers the new mode so output the mode promises never to print is
never buffered, the same reasoning none already had. Pinned byte-exact
(the exact-expected-set rule: a mangled leak sails past not.toContain)
across success/hit/failure/skip, plus parse pins for both spellings
and the typo message naming all four modes.

The comparison doc audit that sourced this also surfaced two stale
cloud references, corrected in place: gap #1 advertised the removed
platform's S3 blob backend (now points at the archived design doc),
and gap #12 claimed the deleted dashboard covers last-run replay —
its removal raises that gap's value, and the entry now says so.
Exelord added a commit that referenced this pull request Aug 25, 2026
Comparison gap #12, closed the wave after its value was re-assessed:
the deleted dashboard had been this gap's answer, and vx last is the
CLI surface that replaces it. Bare = the most recent run (verdict,
command, timing, branch @ sha, CI, counts, per-task table with
failures first); vx last <runId> replays a specific run; --list[=N]
prints recent run ids; --format json emits {invocation, tasks}.

Entirely read-only over the existing metrics layer — zero new SQL, no
config evaluation, no cache probe. The verb slots beside vx why: same
DB-reading grammar, same parser conventions including why's
flag-name-first lesson.

Pinned e2e via bin.ts subprocesses (replay content, --list round-trip,
a FAILED run with failures first, JSON shape, unknown-id fails loud
pointing at --list) plus parser units for both flag spellings and
every rejection message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants