perf(os/mkosi): make cached builds reuse what has not changed

[kvinwang]

Problem

A cached mkosi build repeated work whose inputs had not changed. Profiling a
warm build — every component restored from cache, so the only real work left
was assembly — put 498 s on the clock:

phase	elapsed
mkosi.finalize — squashfs compression	288 s
mkosi.postoutput — release tarballs	66 s
mkosi.build — 9 components, all cache hits	56 s
install Debian rootfs + build packages + metadata sync	70 s

Each has a separate cause, and none of them is the component cache failing:

1. The compressor ran with -processors 1.
2. Two release tarballs, ~58 s of gzip over artifacts already sitting unpacked
   beside them — a redistributable an iteration build does not want.
3. The package install tree was rebuilt from scratch every run.
4. Cache hits still cost 33 s, because the staging trees they extract into
   lived in the per-build sandbox. The kernel archive alone unpacks 6.4 GiB.
5. A source edit rebuilt the entire Rust workspace, dependencies included,
   because CARGO_TARGET_DIR and CARGO_HOME were in that sandbox too.

Fix

Five optimizations, each measured separately, plus the fixes for what they
broke on the way. The shape is the same throughout: work whose inputs have not
changed should not be redone, and anything persisted must carry a key that
actually invalidates it.

change	saves
parallelize the rootfs compressor	285 s
skip release tarballs in cached builds (--archive opts back in)	58 s
reuse the package install tree via Incremental=	69 s
persist the cargo target directory and CARGO_HOME	122 s
persist component staging trees so a hit skips extraction	31 s

Results

path	before	after
unchanged rebuild	498 s	96 s
rebuild after editing one line of Rust	263 s	131 s

Verification

Two gates, both run on the final tree. repro-check's leg a is a cold,
archiving build, so it doubles as the reference for the byte comparison rather
than paying for a separate cold build.

Cached and cold builds agree byte for byte — 15/15 artifacts:

一致 auth_hash.txt aws-pcrs.json bzImage digest.txt disk.raw initramfs.cpio.gz
     measurement.{aws,gcp,snp,tdx}.cbor metadata.json ovmf.fd ovmf-sev.fd
     rootfs.img.parted.verity sha256sum.txt
gate2=PASS

repro-check passes, its two legs deliberately running with different job
counts (32 vs 16) so a parallelism-dependent output fails rather than hides.

This gate ordering matters: repro-check forces cache=0 and therefore never
executes the cached path at all. Every change here is confined to that path, so
repro-check would pass even if the cache were producing garbage. The byte
comparison is the only gate that covers them.

Each new cache also carries a falsification test — a change it could hide must
still be visible in the resulting image:

• editing a file under mkosi.skeleton/ appears in the built rootfs
• switching a component's cache key clears files the previous key had staged,
  and a staging tree deleted behind the stamp's back is re-extracted rather
  than skipped (both pinned in test-dev-cache.sh)

Notes for review

Incremental= could not be enabled as-is. mkosi keys it on &d~&r~&a~&I,
whose specifier set contains no digest of Packages= or mkosi.skeleton/.
Editing either would have left the key untouched and silently restored a stale
tree, so a digest of those inputs is folded into CacheKey=.

Persisting CARGO_HOME moved the dependency git checkouts, whose paths rustc
embeds in the binaries. Without remapping them the cached build's binaries
differed from the cold build's — and only inside the binaries, with every path
that was remapped looking identical. The remap now applies to both layouts so
they land on the same string; this changes the absolute path embedded in
release binaries from /var/tmp/dstack-cargo-home to /usr/src/dstack-cargo-home,
which is a one-time change to the artifact bytes.

Three commits here fix problems the persistence changes introduced: the kernel
staging tree is a merge target rather than a cache and collides with itself
when persisted; the component work directory holds git checkouts that must
start empty; and the staged marker has to live where the per-build wipe cannot
reach it. They are kept as separate commits because each documents a distinct
reason a directory is or is not safe to persist.

Not achieved: the 60 s target for an edited rebuild. 131 s breaks down as
cargo 43 s, kernel re-extraction 17 s, and 68 s of fixed cost — finalize's
squashfs and verity, postoutput's partitioned image, four measurements and the
UKI, plus mkosi's own startup. Even with cargo and the kernel at zero that
fixed cost exceeds 60 s. Reaching it would mean not producing a complete
measurable image on every iteration, which is a different feature rather than a
further optimization.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.