Download deadlines are flat totals, so a healthy large transfer fails as a timeout instead of classifying

[mobileskyfi]

Promoted 2026-07-31 — this is now B13 of #110, a first-class bite, and the recommended next one. It was filed as follow-up from B3; it is not a side note any more. Four other bites are wrong or unmeasurable until it lands (see "Why this blocks the program"). Raised to P1 for that reason, not because anything new broke.

Split out of B3 of #110 (#104), where a genuinely cold cache first made this reachable. The cache-key work does not need this — B3 only stopped a test from calling a completed download a failure. This issue is the download path itself.

The defect

Both download paths bound a transfer by total duration, with a number that has no measurement behind it — and they do not even agree on that:

• src/lib/images.ts:50 — fetchResilient(url, { signal: AbortSignal.timeout(120_000) }), per attempt, 3 attempts with backoff.
• src/lib/packages.ts — downloadPackages() has no deadline and no retries at all; the only bound was the caller test's flat 120_000.

A total-duration deadline cannot distinguish "slow" from "stuck". It fires on a healthy transfer whose only sin is being large, and on the retry path it then re-downloads from zero — turning one slow transfer into three.

Evidence (two independent measurements)

Run 30606079288, hosted linux-x86, first cold cache after the #104 key redesign:

artifact	size	cold time	throughput
all_packages-x86-7.22.1.zip	9.8 MB	16.2 s	~0.60 MB/s
all_packages-arm64-7.22.1.zip	52.2 MB	exceeded 120 s	~0.35 MB/s

The arm64 transfer completed — the next test in the file logged Using cached packages: 7.22.1 (arm64). The failure was (fail) … timed out after 120000ms: bounded, but naming nothing about downloads, sizes or throughput.

B7's local full-suite run (test/lab/full-suite-resource-trend/REPORT.md) is the same mechanism on the image path: provisioning.test.ts spent 619 s of 992 s with zero QEMU processes alive, downloading 41.5 MB images that "needed two retries each". At 120 s per attempt and ~0.35 MB/s, a 41.5 MB image is right at the edge — so the retries were the deadline firing on healthy transfers, not on failures.

Why this blocks the program

One broken deadline currently propagates into four other bites of #110:

• B4 (per-file watchdog, CI: evidence-preserving per-file watchdog and incremental run ledger #77) — its caps come from a ci-data window that contains retry-inflated healthy files. A 992 s provisioning.test.ts is 619 s of download. Caps derived today are too high, in the one direction a watchdog cannot afford, since its whole job is to fire before the ~40–50 min runner-loss boundary. And a 15-minute cap would falsely kill a healthy transfer on a cold cache.
• B8a/B8b (CI: macos-x86 full suite wedges at position 10 in provisioning.test.ts — host-level runner loss, mechanism unexplained (HVF) #76) — macos-x86 has had no cache writer since ci: key the CHR image cache by resolved version, with one owner (B3 of #110) #117 (the save step has no if: always(), so a leg that never finishes never saves). It pays a full cold download every run, and today that cost is a retry-count lottery rather than a measurement.
• B11 (CI: retune oversized boot envelopes from path-specific timing data without losing forensic headroom #106) — a download sample would report the retry count, not the path's cost.
• B12 (CI: retune oversized boot envelopes from path-specific timing data without losing forensic headroom #106) — this is exactly the accidental slack that retune exists to remove; and a download budget must sit in its own partial order (download stall + transfer deadline < test timeout), not nested inside the boot envelope.

Proposed approach

1. Two deadlines, not one — and report which fired

The original stall-only proposal has a hole: a transfer trickling at 1 byte/s resets its stall deadline forever and never aborts. Use both bounds:

1. A resettable stall deadline — reset on each received chunk. A moving transfer is never aborted for being slow; a wedged one fails in seconds rather than minutes.
2. An outer transfer budget derived from content-length and a named floor throughput, so a trickle still terminates. COLD_DOWNLOAD_FLOOR_BYTES_PER_S (test/integration/timeouts.ts:79 — 120 000 B/s, about a third of the slowest observed 0.35 MB/s) is the constant to derive from. Item 4 below already says the two must not drift; the way to guarantee that is one home for the constant, imported by both, not a copy. Where content-length is absent, fall back to a stated total cap and say so in the outcome.

2. A named outcome

A download that gives up should carry bytes received vs content-length, elapsed, observed throughput, and which of the two deadlines fired — so a CI log answers "was this slow, stalled, or refused?" without re-running it. DOWNLOAD_FAILED exists; a distinct stall classification is warranted. B11 samples this shape, so it matters beyond this issue, and #110's infra-download outcome class needs the same sub-structure: "stalled" and "slow past its transfer budget" are different findings and only one is an infrastructure problem.

3. One path, not two

images.ts and packages.ts share the download helper; the retry semantics are stated in one place. Today only one has retries and only one has a deadline, which is how the two measurements above came from two different bugs with one cause.

4. Carry the measurement into #106

If this issue produces a better floor number than COLD_DOWNLOAD_FLOOR_BYTES_PER_S, that constant follows it rather than the two drifting apart.

Done-when

• A slow-but-moving download of the 52.2 MB arm64 zip completes rather than aborting, on a link at the observed ~0.35 MB/s.
• A transfer trickling below the floor throughput terminates against the outer budget rather than resetting its stall deadline forever.
• A wedged transfer fails with a named outcome carrying bytes/expected/elapsed/throughput and which deadline fired, inside its enclosing test and file budget.
• images.ts and packages.ts take the same path; the retry semantics are stated in one place.
• Anchor tests cover all three branches (moving-but-slow, trickle, wedged) against a local stub server — no reliance on download.mikrotik.com being slow.
• One targeted cold-cache dispatch shows a real cold download classified rather than retried.

Out of scope

Cache keys and ownership (#104, landed). Retuning boot envelopes (#106). This issue does not change what is cached, only how a transfer is bounded and reported.

[mobileskyfi]

Promoted to B13 of #110 — a first-class bite and the recommended next one — and raised P2 → P1, as part of the plan revision (comment). Nothing new broke; the dependency is what changed. Four bites are wrong or unmeasurable until this lands:

• B4 (CI: evidence-preserving per-file watchdog and incremental run ledger #77) derives per-file watchdog caps from a ci-data window that contains retry-inflated healthy files — B7 measured provisioning.test.ts at 992 s with 619 s of zero QEMU processes alive. That pushes caps up, the one direction a watchdog cannot afford. And the 15-min cap B4 proposes would falsely kill a healthy cold-cache transfer.
• B8a/B8b (CI: macos-x86 full suite wedges at position 10 in provisioning.test.ts — host-level runner loss, mechanism unexplained (HVF) #76) — macos-x86 has had no cache writer since ci: key the CHR image cache by resolved version, with one owner (B3 of #110) #117, so it downloads cold on every run; today that cost is a retry-count lottery.
• B11 (CI: retune oversized boot envelopes from path-specific timing data without losing forensic headroom #106) would sample the retry count, not the path cost.
• B12 (CI: retune oversized boot envelopes from path-specific timing data without losing forensic headroom #106) exists to remove exactly this accidental slack.

One design refinement before building it. The stall-only proposal has a hole: a transfer trickling at 1 byte/s resets its stall deadline forever and never aborts. The body now specifies both bounds —

1. a resettable stall deadline (a moving transfer is never aborted for being slow), and
2. an outer transfer budget derived from content-length and a named floor throughput, so a trickle still terminates; fall back to a stated total cap when content-length is absent, and say so in the outcome.

The failing outcome reports bytes/expected/elapsed/throughput and which of the two deadlines fired — that distinction matters past this issue, since #110's infra-download class needs the same sub-structure ("stalled" and "slow past budget" are different findings, and only one is infrastructure).

On item 4: COLD_DOWNLOAD_FLOOR_BYTES_PER_S should get one home imported by both call sites, not a copy. Asking two constants not to drift is how they drift.