feat(core): add --max-cache-ram bounded asset-discovery cache [PER-7795]

[pranavz28]

Summary

Adds a --max-cache-ram <MB> flag (plus PERCY_MAX_CACHE_RAM env and discovery.maxCacheRam percyrc) to cap the asset-discovery cache memory. When set, a hand-rolled byte-budget LRU evicts least-recently-used resources to stay within the cap. Evicted resources are not dropped — they spill to a per-run disk tier and are transparently rehydrated on cache lookup, so a memory-bounded run behaves like an unbounded one at the cost of a local disk read (cheaper than an origin refetch). When unset, behavior is identical to today plus a one-shot warn-level log at 500 MB pointing users at the flag before they OOM. Two CLI-side telemetry events (cache_eviction_started, cache_summary) flow through the existing UDP pager → Amplitude pipeline for adoption / hit-rate / disk-tier analytics.

Ships entirely in percy/cli. Zero Percy API changes. Zero new npm dependencies (disk tier uses only Node built-ins: fs, os, path, crypto). @percy/core Node engine unchanged (>=14).

Ticket & planning

• Jira: https://browserstack.atlassian.net/browse/PER-7795
• Task Brief + engineering brainstorm captured in the companion docs (TB, engineering requirements, plan). All five PoCs executed during brainstorming; R6 (server-side OOM-suspected bucket) was dropped after PoC 3 confirmed Percy API does not persist send-events extra and cilogs do not carry CLI process exit codes.

Key decisions

• Hand-rolled ByteLRU (~60 LOC, zero dep) instead of lru-cache npm — v7 is unmaintained-since-2022, v10+ requires a Node engine bump that would break existing users on Node 14/16. Percy's cache usage is ~5% of any LRU library's surface.
• Disk-spill overflow tier (DiskSpillStore) lives in the same cache/byte-lru.js module as the RAM tier — one cache module, two tiers. When RAM evicts, the full resource is written synchronously to a per-run temp directory under os.tmpdir()/percy-cache-<pid>-<rand>/. A slim metadata reference stays in an in-memory Map. getResource() (via the extracted lookupCacheResource helper) falls through RAM miss → disk-index lookup → readFileSync → return full resource. Browser never sees a refetch on a disk hit. On any disk I/O failure (init, write, read) we fall back to the old drop-on-evict behavior — the browser refetches from origin exactly as it would without spill. Index is self-healing (read failure purges the entry). Temp dir is rm'd in the queue 'end' handler; cleanup errors are swallowed so they cannot fail percy.stop().
• Counter-based filenames (dir/1, dir/2, …) rather than URL-derived — no user-controlled data flows into path.join, eliminating the path-traversal surface semgrep flags on URL-in-filename patterns. Collisions are impossible within a run because the counter is monotonic + the dir is fresh per run.
• Sync fs ops on both hot paths, not async. ByteLRU.set is synchronous and getResource is the sync callback CDP network-intercept calls — neither can yield to the event loop mid-operation. Per-entry size is capped at 25 MB upstream in network.js, so worst-case disk latency is a single bounded I/O op — still strictly cheaper than an origin refetch.
• Byte accounting = cache body bytes only (resource.content.length + 512 B per-entry overhead). Flag caps cache, not RSS. MB is decimal (1,000,000 B), not binary MiB. Docs note: real-world RSS is typically 1.5–2× cache bytes due to Node's Buffer slab allocator (PoC 4 calibration).
• MB semantics throughout: --max-cache-ram 300 means 300 MB. Users never write unit suffixes.
• Sub-25 MB values clamp to 25 MB with a warn log (not a hard error). The per-resource ceiling is 25 MB, so any cap below that would silently skip every resource; clamping preserves the user's intent without killing the build. User's original percy.config.discovery.maxCacheRam is NOT mutated — the effective cap lives on percy[CACHE_STATS_KEY].effectiveMaxCacheRamMB and is what cache_summary telemetry reports.
• Warning-at-threshold at warn level so it surfaces under --quiet; suppressed under --silent. Threshold default 500 MB, override via PERCY_CACHE_WARN_THRESHOLD_BYTES (read once at queue construction; debug-logged at startup when the override is active).
• Telemetry folded into two events (not three). cache_budget_configured was dropped because it would fire before percy.build.id exists; its fields live inside cache_eviction_started + cache_summary which are guaranteed to fire post-build-creation. cache_summary is ordered AFTER sendBuildLogs in percy.stop()'s finally so analytics latency cannot delay the primary log egress. cache_summary now also carries eight disk-tier fields (disk_spill_enabled, disk_spilled_count, disk_restored_count, disk_spill_failures, disk_read_failures, disk_peak_bytes, disk_final_bytes, disk_final_entries) so the dashboard can distinguish "disabled", "enabled with no activity", and "enabled with failures".

Commit structure (bisectable)

bf6b92e0 feat(core): spill evicted resources to disk, restore on lookup
a299cd1b test(core): close remaining coverage gaps
e1d6e1c2 fix(core): address PR review (frozen counter, reorder checks, reorder egress, effective cap)
dad8a08e fix(core): clamp --max-cache-ram below 25MB instead of failing
6ae0659b docs(core): document --max-cache-ram flag
d1cf2c0d test(core): integration coverage for --max-cache-ram
2e26f1a5 feat(core): emit cache_eviction_started + cache_summary telemetry
ff45eb38 feat(core): add warning-at-threshold when --max-cache-ram is unset
9e173ecc feat(core): swap Map for ByteLRU when --max-cache-ram is set
0e5364c3 feat(core): extend discovery config schema with maxCacheRam
eed9c323 feat(cli-command): add --max-cache-ram flag (MB units)
537b336f feat(core): add ByteLRU + entrySize helpers

Commits 9 and 10 were added during self-review: clamp behaviour + a separately-broken help-text fixture in cli-command/test/flags.test.js that was stale since the flag was introduced. Commit 12 is the disk-spill upgrade as a single squashed commit — DiskSpillStore, wiring, lookupCacheResource extraction, telemetry fields, and all 15 new test specs land together.

Testing

Automated

• 44 unit specs in packages/core/test/unit/byte-lru.test.js cover ByteLRU, entrySize, DiskSpillStore, createSpillDir, and lookupCacheResource:
  • ByteLRU: unbounded mode, LRU eviction (single + multi), recency bump, oversized entries (skip + preserve prior on re-set), onEvict signature (key, reason, value), .values(), .clear(), .delete() byte accounting, peak-bytes transient high-water, hits/misses/evictions tracking, NaN/negative guards, churn stability.
  • DiskSpillStore: mkdir success + failure (via /dev/null/…), not-ready short-circuit, not-ready destroy no-op, binary round-trip with non-ASCII bytes, string→Buffer coercion, coercion failure (Symbol content), null/undefined content guards, metadata carry-through (sha, root, widths), counter-based filenames, accounting on replace/delete, peak tracking, write failure via mocked EACCES, read self-heal via mocked ENOENT, unlinkSync error tolerance on both delete + overwrite, destroy error swallowing via mocked EBUSY.
  • createSpillDir: uniqueness, percy-cache- prefix, tmpdir scoping.
  • lookupCacheResource: snapshot-local hit, RAM hit, disk hit (with debug-log verification), full miss, disk-present-no-hit, array-root width match, array-root fallback.
• 8 integration specs in packages/core/test/discovery.test.js (describe with --max-cache-ram disk-spill tier): DiskSpillStore presence only when cap is set, spill-on-eviction with cache spill: debug log, byte-for-byte rehydration of binary content, disk-write failure fallback to drop with cache evict: debug log (ENOSPC simulated), saveResource clearing stale disk entries, queue-end teardown (asserts both disk.ready === false and fs.existsSync(dir) === false for race safety), graceful handling of a DiskSpillStore that fails to initialize (via mocked mkdirSync).
• Existing 19 byte-lru unit specs + 13 max-cache-ram integration specs stay green; only the two log-string assertions changed (Skipping cache for resource → cache skip (oversize):, eviction-active info log now mentions "spilling to disk").
• Semgrep: 0 findings on all changed files (ran locally with semgrep --config=auto and the OSS path-join-resolve-traversal rule that previously flagged a path.join(os.tmpdir(), ...) false-positive on createSpillDir; resolved by dropping the unused prefix parameter and hard-coding percy-cache-).
• Local lint on this machine is broken (pre-existing @babel/eslint-parser config detection issue that also reproduces on master); CI will gate.

Security

The 34 Dependabot alerts on this branch are pre-existing transitive dependencies on master (basic-ftp, flatted, ip, lodash, minimatch, rollup, systeminformation, tar, axios, follow-redirects, js-yaml, picomatch, qs, yaml, brace-expansion, @tootallnate/once, tmp, uuid, etc.). This PR introduces zero new npm dependencies. Dependency-bump PRs should be handled separately.

Pending manual verification

The original 4 builds (#353–#356) exercise the max-cache-ram plumbing but pre-date disk-spill. Re-running the 30 × 1 MB heavy-assets workload from the original verification (the one that produced build #361's cache evict: storm) will now show cache spill: + cache disk-hit: lines instead of drop-and-refetch. Build IDs + MCP verification will be added to this PR body once the new builds are produced locally.

Post-Deploy Monitoring & Validation

• What to monitor/search
  • Amplitude (primary): events cache_eviction_started, cache_summary. Track presence, frequency, and field distributions — now including disk_spill_enabled, disk_spilled_count, disk_restored_count, disk_spill_failures, disk_read_failures, disk_peak_bytes, disk_final_bytes, disk_final_entries alongside the original RAM-tier fields.
  • Honeycomb (secondary): service.name=cli traces with core:discovery log namespace. Search for message fragments cache spill:, cache evict: (now only fires when disk spill failed — its presence is a signal, not routine), cache disk-hit:, Cache eviction active — cap reached, oldest entries spilling to disk, Percy cache is using, --max-cache-ram=, is below the 25MB minimum, PERCY_CACHE_WARN_THRESHOLD_BYTES override active, disk-spill init failed, disk-spill write failed, disk-spill read failed, disk-spill cleanup failed.
  • Support channel #percy-support: watch for tickets mentioning max-cache-ram, OOM, heap, killed, /tmp, ENOSPC, disk full, permission denied after release.
• Validation checks (queries / commands)
  • Amplitude filter: event_type IN ('cache_eviction_started','cache_summary') AND cli_version >= <release> — expect non-zero rows within 24 h of release for early adopters.
  • cache_summary.disk_spill_failures / (disk_spilled_count + disk_spill_failures) should be near zero. A rising ratio signals a disk-tier regression (permissions, noexec /tmp, full volume).
  • cache_summary.disk_restored_count > 0 alongside cache_summary.evictions > 0 confirms the round-trip is working in anger, not just the spill side.
  • Support-query: -max-cache-ram across Zendesk/Intercom — expect no new tickets tied to flag parsing, cap enforcement, sub-25 MB clamp, or disk-tier cleanup (leaked temp dirs).
• Expected healthy behavior
  • ≥ 10% of opt-in runs show cache_summary with a non-null cache_budget_ram_mb within 60 days.
  • On runs with evictions > 0: disk_spilled_count should track evictions (1:1 minus a tiny tail of simultaneous-write races).
  • disk_restored_count / disk_spilled_count > 0.5 on typical memory-constrained workloads — proves the disk tier earns its keep.
  • Build-failure rate unchanged vs. pre-release baseline.
• Failure signal(s) / rollback trigger
  • Trigger A: > 10 support tickets in 7 days tied to --max-cache-ram flag parsing, cap clamp, disk-tier init, or "my build started failing after setting this".
  • Trigger B: measurable spike in Percy build-failure rate correlated with CLI versions that include this PR.
  • Trigger C: cache_summary events are never received in Amplitude (pipeline regression).
  • Trigger D: cache_summary.disk_spill_failures > disk_spilled_count across a large cohort — means the disk tier is failing open for most users; the feature would be a net loss over the old drop-on-evict behaviour.
  • Trigger E: cache_summary.peak_bytes values clustered at the 500 MB default threshold (signals the Map-mode counter has regressed to frozen behavior).
  • Immediate action on any trigger: release a patch reverting this PR; tell users to unset the flag as a workaround (flag is opt-in, unset = previous behavior).
• Validation window & owner
  • Window: 14 days post-GA release.
  • Owner: @pranavz28 (Pranav Zinzurde).

🤖 Generated with Claude Opus 4.7 (1M context, extended thinking) via Claude Code + Compound Engineering v2.50.0

[pranavz28]

PER-7795 Disk-Spill — Real-Build Verification Report

Branch: feat/per-7795-max-cache-ram
Verification run: 2026-04-24

All runs below used a local web-t token against test-pranav-8a4f5725 and
the two local origin servers built for this verification:

Server	Port	Description
heavy-assets-server.mjs	9200	30 × 1 MB CSS, no delay (TC1, TC2)
heavy-assets-slow.mjs	9201	60 × 1 MB CSS, 150 ms delay each (TC3, TC4, TC6, TC8)

Test case matrix

#	Case	Flag	Fixture	Build #	Result
1	Baseline (no flag)	(none)	heavy @ 9200	#392	PASS — no spill dir created, no disk telemetry
2	Aggressive spill	--max-cache-ram 1 (→25 MB clamp)	heavy @ 9200	#394	PASS — 7 spills, Cache eviction active info log fired once
3	Live disk-dir inspection	--max-cache-ram 1	slow @ 9201	#397	PASS — peaked at 37 files / 35 MB on disk; byte sizes match spilled resources
4	Disk restore (concurrency:1)	--max-cache-ram 1, discovery.concurrency: 1	slow @ 9201	#401	PASS — 97 spills, 96 cache disk-hit log lines
5	Cleanup on stop	—	—	—	PASS — $(os.tmpdir())/percy-cache-* removed after every run
6	Telemetry payload	--max-cache-ram 1, concurrency: 1	slow @ 9201	#403	PASS after fix — all 8 disk_* fields populated
7	End-to-end build health	verified on #392, #403, #405	—	—	PASS — every completed comparison has diff ratio 0.0000 vs baseline
8	Moderate cap	--max-cache-ram 40, concurrency: 1	slow @ 9201	#405	PASS — 82 spills, 81 disk-hits, disk peaked at 22 entries / 21 MB

Specific resource → disk-file evidence (TC3, live)

Process PID 27680, spill directory
/var/folders/4g/nq932q454bjbwx6xnp_zprdh0000gn/T/percy-cache-27680-d47eaa7d,
captured 4.8 s into the run:

-rw-r--r-- 1  3060 bytes    1   ← http://127.0.0.1:9201/          (3,060-byte HTML)
-rw-r--r-- 1  1000000 bytes  2   ← http://127.0.0.1:9201/slow-0.css
-rw-r--r-- 1  1000000 bytes  3   ← http://127.0.0.1:9201/slow-1.css
-rw-r--r-- 1  1000000 bytes  4   ← http://127.0.0.1:9201/slow-3.css
-rw-r--r-- 1  1000000 bytes  5   ← http://127.0.0.1:9201/slow-2.css
-rw-r--r-- 1  1000000 bytes  6   ← http://127.0.0.1:9201/slow-4.css
-rw-r--r-- 1  1000000 bytes  7   ← http://127.0.0.1:9201/slow-5.css

Filenames are the DiskSpillStore monotonic counter (1, 2, 3, …), and
sizes match each resource exactly (3 060 B for the HTML, 1 000 000 B for
each CSS). By later iterations the directory held 37 files totalling
~35 MB. After percy.stop() the directory was gone.

Telemetry payload (TC6, temporary probe)

With a one-line debug probe added inside sendCacheSummary and reverted
before commit, the payload shipped to Percy was:

{
  "cache_budget_ram_mb": 25,
  "hits": 24,
  "misses": 220,
  "evictions": 97,
  "peak_bytes": 25016372,
  "final_bytes": 24015860,
  "entry_count": 25,
  "oversize_skipped": 0,
  "disk_spill_enabled": true,
  "disk_spilled_count": 97,
  "disk_restored_count": 96,
  "disk_spill_failures": 0,
  "disk_read_failures": 0,
  "disk_peak_bytes": 36003060,
  "disk_final_bytes": 36003060,
  "disk_final_entries": 37
}

All eight disk_* fields are populated and match the discovery log.

Bug found and fixed during verification

The first TC6 run produced disk_spill_enabled: false with every
disk_* field zeroed, even though the log showed 97 spills and 96
disk-hits. Root cause: percy.stop() calls discovery.end() (which runs
diskStore.destroy() and delete percy[DISK_SPILL_KEY]) before
sendCacheSummary() reads those references.

Fix (commit 2422b01d): the end handler now snapshots
diskStore.stats (plus ready) onto stats.finalDiskStats before
destroy(); sendCacheSummary prefers the live store and falls back to
the snapshot, so both in-discovery tests and post-discovery real builds
populate the telemetry correctly. Two new specs cover each branch.

Commands reproduced here for reference

# TC1 — baseline
zsh -ic 'web-t && node packages/cli/bin/run.cjs snapshot \
  /tmp/heavy-snapshots.yml --verbose'

# TC2 — aggressive spill (clamped to 25 MB)
zsh -ic 'web-t && node packages/cli/bin/run.cjs snapshot \
  --max-cache-ram 1 /tmp/heavy-snapshots.yml --verbose'

# TC4 / TC6 — serial disk restore with full telemetry
zsh -ic 'web-t && node packages/cli/bin/run.cjs snapshot \
  -c /tmp/percy-serial.yml --max-cache-ram 1 \
  /tmp/slow-multi-snapshot.yml --verbose'

# TC8 — 40 MB moderate cap
zsh -ic 'web-t && node packages/cli/bin/run.cjs snapshot \
  -c /tmp/percy-serial.yml --max-cache-ram 40 \
  /tmp/slow-multi-snapshot.yml --verbose'

/tmp/percy-serial.yml:

version: 2
discovery:
  concurrency: 1