Stop reinstalling rootfs and initramfs on every sdk install

[mobileoverlord]

Problem

avocado sdk install redid the rootfs and initramfs installs in full on
every invocation, even when nothing had changed. Two independent defects
caused it.

1. False "packages removed" detection wiped both sysroots every run.
detect_sysroot_package_removals compared config-declared package
names against locked names and returned true — meaning rm -rf $AVOCADO_PREFIX/<sysroot> plus a from-scratch reinstall — whenever the
lock held names the config didn't.

But the install auto-appends two packages the config never declares, then
records them in the lock:

• packagegroup-avocado-{rootfs,initramfs}-modules-<kver>
• kernel-image-<kver> (rootfs only)

From the second run onward those always read as removed. Because
get_rootfs_packages / get_initramfs_packages default to the single
meta-package, this fired on every project, not only ones overriding
rootfs.packages. Confirmed against every real lockfile in the tree —
for example references/jetson-trt/avocado.lock:

	locked names	config names
rootfs	avocado-pkg-rootfs, kernel-image-6.8.12-…, packagegroup-avocado-rootfs-modules-6.8.12-…	avocado-pkg-rootfs
initramfs	avocado-pkg-initramfs, packagegroup-avocado-initramfs-modules-6.8.12-…	avocado-pkg-initramfs

Second-order effect: the same path called
remove_packages_from_sysroot, dropping those packages' version pins
from the in-memory lock. build_package_spec_with_lock then fell back to
a bare package name, so dnf resolved the kernel image and module
packagegroup to newest available instead of the pinned NVR.
avocado.lock looked pinned on disk (it is rewritten post-install) but
the pin never applied.

2. The install stamps were written but never read. install_sysroot
wrote rootfs/install.stamp and initramfs/install.stamp, but nothing
consulted them to skip work — across the whole CLI, stamps were only
prerequisite gates. So even with defect 1 fixed, each run still paid a
kernel repoquery, off-kernel exclude computation, a repoquery existence
probe, the dnf transaction, query_installed_packages, kernel-sysroot
staging, and the stamp write.

Solution

An unchanged config now does zero container work for rootfs/initramfs; a
changed one reinstalls exactly the sysroot(s) affected; avocado.lock
pins actually bind.

Key changes

• Compare removals against the effective set. install_sysroot
  resolves the kernel and computes the auto-appended names before removal
  detection, and detect_sysroot_package_removals now takes the effective
  name set (config packages ∪ auto-appends) and returns the removed names
  rather than a bool. That set is built once and reused as the post-install
  lock query list, replacing a second, duplicate computation. A kver change
  falls out of the same mechanism, so the dedicated kernel-pin-change block
  became the reason for the clean rather than a second code path, and the
  two verbatim inline rm -rf RunConfig blocks collapse into one
  clean_sysroot helper built from the existing clean_sysroot_command.

• Skip the install when the stamp is current. New
  read_stamps_batch / StampBatch in utils/prerequisites.rs wraps
  generate_batch_read_stamps_script + one container run + parsing. sdk install reads both sysroot stamps in a single invocation before the
  parallel phase and passes them in via SysrootInstallParams; the
  standalone rootfs install / initramfs install read their own through
  the same helper. install_sysroot short-circuits before any container
  call. Stamps live in the /opt/_avocado named volume rather than a host
  bind mount, so a container run is unavoidable — hence batching.
  check_prerequisites and five sites that hand-rolled the same RunConfig
  (build.rs, sdk/{clean,package}.rs, ext/{clean,checkout}.rs) are
  ported to the helper.

• Make the hash trustworthy enough to skip on. The two per-sysroot
  hash functions collapse into one core that now hashes the effective
  package map (so an absent rootfs.packages and an explicit default hash
  identically), plus sdk.repo_url, sdk.repo_release — the hook that
  makes a snapshot bump or avocado update land — and
  sdk.disable_weak_dependencies. StampInputs.package_list_hash is
  populated with a digest of that sysroot's lockfile pins, so a hand-edited
  avocado.lock or an avocado unlock goes stale automatically. An empty
  pin set hashes to its own value rather than to None, because
  Stamp::is_current only compares that field when both sides carry one —
  otherwise a cleared lock would compare equal by omission.
  STAMP_VERSION is deliberately not bumped: that would invalidate every
  SDK/ext/runtime stamp. Changing only these two hash bodies makes existing
  rootfs/initramfs stamps mismatch naturally, costing one reinstall.

• Make clean invalidate its stamp. clean_sysroot_command also
  removes $AVOCADO_PREFIX/.stamps/<sysroot>, mirroring ext clean.
  Without it, rootfs clean && sdk install would skip against a current
  stamp and leave an empty sysroot.

• Drop the mid-flight lockfile saves. install_sysroot and
  stage_kernel_sysroot_from_rootfs saved their lockfile clone while the
  rootfs and initramfs tasks ran concurrently on separate clones —
  last-writer-wins, healed only by the caller's final save. Saving is now
  the caller's job throughout, which removes the race.

Escape hatches are unchanged in spirit: --no-stamps disables the read
along with the write, so it always reinstalls, and avocado rootfs clean
remains the hard reset.

Reviewer notes

cargo fmt --check, cargo clippy --all-targets -- -D warnings, and the
full cargo test suite are green (1124 lib tests plus the integration
suites, no failures).

New unit tests:

• detect_sysroot_package_removals reports nothing for the exact
  references/jetson-trt/avocado.lock shape — a default config with both
  auto-appended kernel packages locked. This is the regression test for the
  bug.
• It still reports a genuinely config-dropped package, and reports the
  previous kver's auto-appends as stale after a repin.
• The rootfs hash is stable across absent-vs-explicit-default
  rootfs.packages, and moves on an added package, on repo_url,
  repo_release and disable_weak_dependencies, and on a lock re-pin.
• package_list_hash is order-independent (pins come out of a HashMap),
  and an avocado unlock invalidates is_current.
• A rootfs-only config edit does not invalidate the initramfs hash.

End-to-end verification against a live feed is in progress separately.

[Copilot]

Pull request overview

This PR optimizes avocado sdk install by preventing unnecessary rootfs/initramfs reinstalls when inputs haven’t changed, and by fixing erroneous “packages removed” detection that caused sysroot wipes and ineffective lockfile pinning.

Changes:

• Fix sysroot removal detection to compare against the effective package set (config packages plus auto-appended kernel/module packages), returning concrete removed names.
• Add batched stamp reading (read_stamps_batch / StampBatch) and use sysroot install stamps to short-circuit rootfs/initramfs installs when current.
• Strengthen sysroot install stamp hashing to include effective packages, SDK feed identity/resolver flags, and a digest of lockfile pins; update clean to invalidate install stamps.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/utils/stamps.rs	Adds sysroot stamp input plumbing (effective package hashing + lock pin digest) and extends hash coverage; updates/expands unit tests.
src/utils/prerequisites.rs	Introduces StampBatch + read_stamps_batch to run one container invocation for multiple stamp reads (local or --runs-on).
src/commands/sdk/package.rs	Switches SDK prerequisite stamp checking to read_stamps_batch.
src/commands/sdk/install.rs	Prefetches rootfs/initramfs install stamps in one batched read and passes them into sysroot install to enable skipping.
src/commands/sdk/clean.rs	Switches SDK prerequisite stamp checking to read_stamps_batch.
src/commands/rootfs/install.rs	Implements sysroot install skipping via prefetched stamp + stronger inputs; fixes removal detection; refactors clean; adjusts lock saving responsibility.
src/commands/rootfs/clean.rs	Ensures clean removes the sysroot’s install stamp directory as well as the sysroot.
src/commands/initramfs/install.rs	Adds prefetched install-stamp read and moves lock saving responsibility to the command.
src/commands/ext/clean.rs	Switches SDK prerequisite stamp checking to read_stamps_batch.
src/commands/ext/checkout.rs	Switches stamp checking to read_stamps_batch for prerequisites.
src/commands/build.rs	Switches prerequisite stamp checking to read_stamps_batch.
CHANGELOG.md	Documents the fixed reinstall behavior, stamp-based skipping, and lock pin correctness.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mobileoverlord]

@jetm this is ready for review.

All four review threads are addressed and resolved:

• Both Copilot lockfile threads — replied with the trace rather than a change. The stamp write is the last fallible step before Ok(()), so the lock and the stamp are now persisted under the same condition and cannot disagree. The state Copilot describes is reachable but benign, because no stamp is written either, so the next run reinstalls instead of trusting stale pins. This is strictly tighter than main, which saved the lock at rootfs/install.rs:791 — before kernel staging and before the stamp write — leaving a persisted lock from a run that never completed.
• @nicksinas's verbose nit — fixed in b9dc588. Dropped verbose: self.verbose from all three new stamp-read RunConfigs so they match the five pre-existing read_stamps_batch callers.
• @nicksinas's let-chain nit — declined with a reason: this crate is edition = "2021" and let-chains are gated to edition 2024, so the suggested form doesn't compile (verified against rustc 1.94). Worth revisiting on an edition bump.

Verified locally on the pushed tip: cargo fmt --all -- --check clean, cargo clippy --all-targets --all-features -- -D warnings clean, and cargo test --all green at 1124 + 1132 + 126 + 20 + 5 + 49 + 6 + 7 + 6, zero failures.

One thing worth a second opinion, since it's the actual behavioral heart of the PR: the skip short-circuit means an unchanged project does no container work for rootfs/initramfs. The escape hatches are --no-stamps and avocado {rootfs,initramfs} clean. If you can think of an input that should invalidate the stamp but isn't in compute_install_stamp_inputs, that's the bug worth finding here.

[jetm]

Reviewed this at b9dc588 and built it in a scratch worktree - cargo test --no-run exits 0. Twelve findings inline, most severe first. The findings themselves are code-traced rather than build-derived, except where a comment says otherwise.

The one that matters most is the first: the sdk install lock merge copies entry.rootfs but not cleared_sections, and because this PR also removed the save() that used to run on the clone (base install.rs:791), a kernel-pin-change clean is no longer authoritative on disk. The result is that avocado.lock permanently regains the old kver's keys, so every later run reads stale and wipes both sysroots - which is the exact loop this PR exists to remove, on the path it is named for.

A pattern worth noting across findings 3, 4, 7 and 9: moving the lock save after the stamp block introduced several windows where an install fully succeeds on disk but its pins or its teardown are skipped. Individually each is a narrow failure; together they mean a partial failure now tends to leave the stamp claiming success while the state behind it is wrong, and the recovery the error text names ("run rootfs install") is the command that then answers "up to date" and exits 0.

Two of the twelve I marked lower-confidence rather than dropping: the latched failed wipe (I could not stage a forced non-zero exit from the clean invocation) and the sdk.image omission from the sysroot stamp (I could not name two images that yield different sysroot contents for identical feed plus config, so treat it as a question about which inputs belong in that hash).

Things I checked and decided are NOT problems, so nobody re-derives them: avocado build --runs-on reading stamps from the local volume is byte-identical to pre-PR behaviour and purely pre-existing; the rm -rf "$AVOCADO_PREFIX/..." shell is not an injection risk because sysroot_dir is a compile-time literal and the prefix is exported unconditionally by the entrypoint; and STAMP_VERSION not being bumped is safe in this direction, since an old stamp cannot match the wider hash and so reads stale.

[mobileoverlord]

@jetm all twelve are addressed in e052539 (three commits), with per-thread replies inline. Short version:

Fixed as reported (9): the cleared_sections merge drop, sdk clean leaving the install stamps, the resurrected pin in remove_packages_from_sysroot, the two ? sites skipping teardown, the stamp surviving a failed kernel staging, the stamp-write window discarding lock pins, the transient repoquery wipe, --dnf-args missing from the stamp inputs, and the vacuous first-install test.

Both lower-confidence ones were real (2). The latched failed wipe I fixed on reasoning rather than a reproduction, same as you — clean_sysroot now reports its outcome instead of .ok()-ing it. On sdk.image: I think your uncertainty was aimed at the wrong thing. You couldn't name a pair of images producing different sysroots for identical feed plus config, and neither can I, but the defect doesn't need one — the two hashes disagreeing about what counts as an input is the bug. Folded it in to match compute_sdk_input_hash. I left sdk.container_args out; those are mounts and env for the run rather than the transaction, and I'd want a concrete reason before invalidating both sysroots on a -v change.

Cleanup (1): StampBatch holds only the parsed map; validate_stamps_batch keeps its &str signature for its ~20 callers and delegates to a new validate_stamps_parsed.

Two things worth pulling out of the threads:

The two cleared_sections findings were one bug wearing two hats — a section that is authoritative in memory losing that fact before the disk merge. Propagating across the sdk install merge does nothing for avocado rootfs install, which never goes through it, so both sites genuinely needed fixing. Neither is visible in a lockfile, since the flag is #[serde(skip)].

The failure modes now fall on the harmless side, consistently. A stamp that couldn't be written is benign — its absence makes the next run reinstall. A lock that wasn't saved is not benign — the sysroot moved without it. So the stamp write is now a helper whose failure warns rather than propagating, and the stamp is gated on all three integrity conditions (versions_recorded && kernel_staging_ok && install_is_clean), naming which one failed when it skips. I didn't move the lock save back to its pre-PR position, since that's what the clone design removed it for; making the tail infallible gets the same guarantee without clobbering the parent lock.

One thing found while fixing yours: install_sysroot's doc comment was sitting above package_exists_in_target_repo, so install_sysroot shipped undocumented and the probe carried prose describing a different function. Pre-existing on this branch, not from your review, but the same shape as the misattachment you caught elsewhere. Both reattached.

New tests: removal direction (verified it fails under inverted operands, per the CLAUDE.md rule), probe-output parsing across failed/empty/populated/banner-prefixed, --dnf-args invalidating while an empty list doesn't, and sdk.image invalidating.

On CI: nothing was actually wrong with it. The two red Analyze rows were CodeQL jobs with conclusion cancelled, not failure — killed at 15m02s each in the Aug 6 Actions outage, which gh pr checks renders as fail. The Extension Test workflow had also been stuck queued for 101 hours from the same outage, which is why the board build and version check were missing from the list entirely. Cancelled it and pushed; the full set is running now. Locally fmt --check, clippy --all-targets --all-features -D warnings and the suite are clean. I also merged main in.