Opt-in hard-link/reflink cache restore to avoid writing every cached output twice

[jlapenna]

Summary

Every cached output is written to disk twice, once into the local cache directory and once into the workspace. On ephemeral CI runners, where the local cache is discarded when the job ends, the first copy is almost pure write volume. Could Nx offer an opt-in restore mode that hard-links (or reflinks, where the filesystem supports it) instead of copying?

What happens today (nx 23.2.1)

• Remote hit: remoteCache.retrieve(hash, cacheDirectory) extracts the artifact into <cacheDirectory>/<hash>/, then copyFilesFromCache (native copy_dir_all in packages/nx/src/native/cache/file_ops.rs) copies each file into the workspace.
• Locally executed cacheable task: put() copies the task's outputs into <cacheDirectory>/<hash>/, and remoteCache.store() uploads from there.

Measured: restoring one Node bundle target from a self-hosted remote cache into an empty NX_CACHE_DIRECTORY on Linux/ext4:

• Workspace output: 37 MB.
• Cache directory afterwards: 79 MB, holding separate full copies (link count 1) of the same 38 MB bundle, two of them because two cache entries produce that file.

Why it matters

On our self-hosted CI fleet (ephemeral Docker runners, Nx remote cache), dependency installs are already metadata-only after other fixes. The remaining per-job disk writes are dominated by build outputs that Nx writes twice. Across the fleet that is hundreds of GB per day onto consumer NVMe drives, and SMART wear is climbing measurably. Keeping the cache in RAM isn't an option for memory-bound runners.

Proposal

An opt-in setting, for example NX_CACHE_OUTPUT_MODE=copy|hardlink|reflink|auto (or an nx.json option), used on restore and optionally when storing:

• reflink/auto: use FICLONE/copy_file_range clone semantics where the filesystem supports them (XFS, btrfs, APFS clonefile). Blocks are shared until written, so this is safe by construction.
• hardlink: when the cache directory and the workspace are on the same filesystem, link instead of copying. It's opt-in because a tool that modifies outputs in place would also modify the cache entry; making cache files read-only would turn that into an error instead of silent corruption.
• Default: stays copy.

This touches the same code path as #37090, and #37085 (which parallelizes copy_dir_all); a link or clone mode would compose with that work.

Happy to provide more measurements or test a branch.

[christopher-buss]

+1, with a second use case: the local edit loop, not CI.

Setup: Windows 11 24H2 (build 26100). Workspace and NX_CACHE_DIRECTORY are on the same ReFS volume. Daemon on. One cacheable compile target writes ~12.1k output files. Two more tasks in the same run write ~2.3k files. The compiler is incremental (tsbuildinfo) and changes 1–2 output files per edit.

Problem: every edit is a cache miss, so put() copies the full ~14k-file output set into a new cache entry. The next edit has a new hash, so the entry is never read. The daemon's outputs-hash map helps on hits only.

Measured (median of 3 runs, nx run <target> after a body-only edit in one source file, three edit locations):

	edit 1	edit 2	edit 3
23.3.0-beta.6	21.7 s	17.9 s	24.5 s
23.3.0-beta.8 (#37085)	16.6 s	14.1 s	18.0 s

• beta.6 CPU profile of one edit: ~10.6 s CPU in Cache.put (native put → copy_dir_all). The compiler took 2.3–3.2 s.
• beta.8: the executor takes ~3 s, so ~15 s per edit is spent outside it. I have not profiled beta.8 yet, so I cannot say how much of that is put.

What would help here:

• A reflink/auto store mode that covers Windows: ReFS supports block cloning (FSCTL_DUPLICATE_EXTENTS_TO_FILE).
• Per-file dedupe across entries (content-addressed blobs), so a new entry stores only the files that changed. Today each entry is a full copy.

Happy to test a branch on Windows/ReFS.