v1.13.0

Added

• --readahead-evidence-per-run: the evidence gate can measure the current run instead of
  the handle's lifetime (#407). Experimental,
  off by default. The first arm of a 2×2 that has not been measured yet.

  --readahead-evidence-ratio bounds a committed window by the bytes a handle has read — but
  that figure is cumulative and never reset. On a scattered-but-locally-sequential walk
  every run is 0.5–2 MB, so by the third run the handle has earned a window no single run can
  justify, and it re-places that bet at every re-establishment. An external shape sweep measured
  the consequence across three shapes: 48 chunks prefetched and 2 used on a GRIB .idx
  field sweep, 71/11 on a HEMCO time-slice read, 47/13 on a FITS cutout — one mechanism, three
  findings (#407, #409, #222).

  The arithmetic reproduces their measurement exactly: 11.9 MB consumed × 4 ÷ 8 MiB = 5.69 → 5
  blocks, against a measured peak window of 5. A test asserts that correspondence, so if
  either the arithmetic or the measurement moves, the mechanism has to be re-derived rather than
  assumed.

  A run ends at a seek out of an established pattern, a scattered landing, or a
  de-establishment. A genuine sequential copy never reaches any of those, so its run is the
  whole file and this is a no-op for it — that is the witness three cold-start fixes were
  refuted against (#231), and the reason the reset sits at run boundaries rather than on a timer
  or a read count.

  Necessary and not sufficient, which is why it is a flag and not a fix. The dispatch floor
  is 2 blocks — 17 MiB at the default block size — so a 1 MB run still floors far above what
  it wants. Closing the rest needs sub-block dispatch, which is not built. The one precedent
  for this pair (#222) found each half alone made bytes worse and only both together gave 16×,
  so the halves have to be measurable separately.

• A guard that experimental flags default off. Found by a revert sweep: flipping this flag's
  default to true left the entire suite green. "EXPERIMENTAL" in a flag's help is a promise
  that the behaviour is unmeasured, and a binary that ships it on contradicts its own help where
  no operator can see it. Keyed on the help text rather than a list, so a new experimental flag
  is covered the moment it is written.

• lith_streaming_handles_idle_{1s,5s,30s}: the idle distribution over the readahead
  divisor's own population (#312). An
  instrument, not a mechanism — nothing reads these.

  Because the measurement I asked for could not answer the question. I asked for
  lith_streaming_handles against lith_open_handles, and it came back 49 against 160. That
  says #311's divisor input is working — it excludes 111 descriptors, 69 % of them — and says
  nothing about idleness, because a stream is counted whether it read a microsecond ago or
  a minute ago. Both pre-registered rows on that issue were really about #311.

  A follow-up timeline cell established that the idleness is real: on the dominant mount, 7 of
  12 keys had gaps ≥ 2 s carrying 91 % of demand events, median long gap 5.0 s, and
  72 % of those gaps contained no re-open of the key — so a held descriptor sat through
  them. What it could not establish is magnitude, because the timeline is per key and
  cannot see reads the kernel serves from the shared page cache. These gauges are per
  handle, and see exactly the reads the divisor's input is computed from.

  A distribution rather than a verdict, for a stated reason: #312's mechanism needs an idle
  threshold, and that threshold is a tuning constant with no principled value — too short and a
  reader pausing on compute loses its window, too long and the mechanism does nothing.
  Exporting the shape lets the next measurement choose it. The buckets themselves are
  measured, not chosen: 1 s sits inside a read burst, 5 s straddles the ExtData time-slice
  cadence, 30 s is past any slice.

  One walk of the handle map per scrape, through a single collector rather than four
  GaugeFuncs — with four, one scrape could observe three buckets at one instant and the
  denominator at another, and a stream going idle in between would make the denominator read
  smaller than a bucket it contains.

• Docs: a root-created mount needs a root-run readiness check. Reported from a benchmark
  harness: mountpoint -q as a non-root user cannot stat a root-owned FUSE mount, so it
  reports not mounted while the mount is in fact serving — a readiness loop times out and, in
  their case, three mounts stacked up behind one failed check. lith mounts does not rescue
  you either, for a different reason: it reads a per-user record directory and deliberately
  ignores records from a directory it does not own, so a root-created mount is equally
  invisible to a non-root lith mounts. The rule is the same for both tools, and it was in
  neither the docs nor anyone's head.

  Both directions, because the note cost a measurement in the other one. The first version
  of this warned only about a non-root check against a root-owned mount. A later harness ran
  sudo mountpoint against a user-owned mount, which fails the same way — every cell in
  that sweep sat idle for its full 60 s timeout, and the run initially read as a 1.17×
  speed-up that the reporter then withdrew. The note now says the check and the mount must be
  run by the same user, rather than naming one of the two mismatches.

• lith_fill_seconds{kind}: what a fault costs, as a scrape rather than a judgement
  (#362, condition 2). An operator mounted a
  1.206 TB kraken2 database, ran a tool that classified zero reads in eleven minutes, and
  spent three instances and most of a day establishing that the mount was working
  correctly — it was in #232's regime, a
  serial fault stream at ~141 ms per 4 KiB probe, which lith has no lever to speed up. In
  their words: "the per-fault latency, the fill kind, and the random-vs-sequential
  classification are all things lith knows and I couldn't see."

  Fill first-byte latency is now exported under the same kind label as
  lith_fill_bytes_total. kind="demand" is a fill no prediction covered, which is the
  fault-stream shape: its quantiles answer what is a fault costing me, its _count answers
  how many faults, and its share against kind="whole" answers is this mount streaming or
  faulting. Those are the three things they could not see.

  Labelled by fill kind rather than by detector state, which is what the issue's framing
  suggested. The kind is already decided at every call site of the fetch seam, where the
  prefetch detector's state lives in internal/fuse and would have to be plumbed down through
  the blockstore; and the kind is the more direct answer, because it says what lith did
  rather than what it predicted.

  No threshold and no verdict. Read it with lith_ttfb_floor_seconds, which is
  load-invariant by construction, and the diagnosis reads itself: demand fills at ~141 ms
  against a ~19 ms endpoint floor is one round trip per miss. docs/copy-or-mount.md now
  carries the three queries and how to read them, including the case that is not this
  regime — demand latency far above the floor is queueing, not S3. Two defaults placed from a
  plausible-looking constant have already had to be withdrawn here
  (#340,
  #349), and "is my workload in this regime"
  is a question about the workload.

Changed

• The in-region evidence-gate ratio is 2, not 4 (#407).
  --readahead-evidence-ratio's region-derived default now bounds a committed window to twice
  the bytes a handle has actually read. Over-fetch on the shape the ratio was originally set
  for — #284's single-variable NetCDF-4 read — goes 2.66× → 2.28×; nothing measured got
  worse.

  This is a smaller default, not a new mechanism, and it was decided by a sweep rather than
  by argument. An external 2-vs-4 run across nine shapes found ratio 2 free or better on every
  one: the single var1 reader 59.8 → 51.4 MB, six concurrent readers 358.6 → 308.3 MB
  (2.62× → 2.26×), 0.86× bytes in both repetitions, with ratio 4 reproducing the historical
  figures to 0.1 MB as the control. netcdf4_hyperslab, webdataset, concurrent_handles and
  a single-stream dd came out byte-identical at either value.

  The ratio does not interact with the per-handle divisor. The six-reader cell stayed exactly
  6× the single-reader cell at both ratios, which is the property that makes this one constant
  rather than a constant per concurrency level.

  Ratio 1 is not better than 2, measured identical on all three walk shapes (GRIB, HEMCO,
  FITS): by then every window is pinned at the 2-block dispatch floor — 17 MiB at the default
  block size — and the ratio has stopped being the binding constraint. That residual is
  sub-block dispatch's to close, not this default's. The test records 4, 2 and 1 as the three
  values anyone has measured and fails on any other, so the next person to move it has to
  bring a cell rather than an intuition.

Fixed

• The NFS gateway's readahead divisor counted MOUNTED clients, not reading ones
  (#337). A client that sent one MOUNT held a
  share of the prefetch budget for --client-idle — five minutes — whether or not it ever read
  a byte. Measured locally with no client reading anything: 1 mount → 96 blocks, 8 → 61,
  48 → 10, 246 → the 2-block floor. A 48× collapse from registration alone.

  Measured externally on a real gateway, 47 namespaced clients that mounted and read nothing
  against one streaming client: 1267–1300 MB/s with one client, 519–558 with 48 — 2.4× —
  with byte-identical transfers in every arm (3.777 GB, 458 GETs, prefetch issued = used
  = 3594). Purely window depth, nothing wasted, and invisible to any byte or request counter.
  Eight clients cost nothing; the reporter's own prediction of "under 20 %" failed, and it
  binds at ~15 Gbps, so a fatter NIC can only be worse — and a gateway is the one place a fat
  NIC is normal, because it funnels a whole cluster through one.

  The corrected input was always there. roFS.states carries a per-path lastTouch that
  stateFor updates on every read. The comment that justified counting mounts — "per-op
  client activity is not visible through go-nfs's stateless read path" — is true of go-nfs and
  not true of lith's own state, and that distinction was the whole bug. This is the
  gateway's analogue of #311's
  streamingHandles, and the same shape of one-line input change.

  It still divides. The fix changed the divisor's input, not whether there is a share:
  removing the rationing costs 13.8–22.4× wall with 81–92 % of prefetch evicted unread, and
  #309 is what happens when a total replaces the share.

  Why this direction is safe: a reader must have mounted, so read streams can never exceed
  registered clients, so a window can only get larger. It cannot starve anyone — starvation
  is exactly what #309 did. The opposite risk is bounded by the pressure gate that now defaults
  on for every export (#389); the two compose, and this would have been riskier a week ago.

  New --read-idle (30 s) governs how long a path counts as active. The reported defect is
  fixed for any value, because a client that never reads creates no per-path state at all;
  the window governs a second case — a path a still-mounted client read a while ago — where the
  only other bound is the 8192-entry LRU cap, which over a long run is a worse divisor than the
  client count ever was.

  And the fix nearly shipped dead. It was wired in Serve(), which no test calls, while
  the test harness built its own copy of that construction and silently kept the
  MOUNT-counting fallback — so TestFairShareArithmetic went on asserting the old divisor and
  passing. newGateway now exists so there is one place, the harness goes through it, and that
  test asserts the new semantics: mounting eight clients changes nothing, reading eight paths
  divides the budget eight ways.

• The NFS gateway exported the pressure gate's hold count but not the pressure itself
  (#337). RegisterPrefetchBudget was called
  from internal/fuse and nowhere else, so an export published
  lith_prefetch_pressure_held_total while publishing neither lith_prefetch_pressure — the
  quantity it gates on — nor the two tier figures that make it readable. The counters that
  did appear made the gateway look instrumented, which is why nobody noticed until an
  external cell tried to answer whether the 0.85 band transfers to a gateway and found a
  gateway's peak pressure could not be read at all.

  All four gauges are now registered on both commands. The readahead-window gauges stay
  FUSE-only on purpose: they describe per-handle shares and the gateway rations per client.

  Seventh instance of this project's most repeated shape — a second consumer of shared
  machinery with its own wiring, tests covering only the first (#240, #242, #346, #239, #393,
  #337). So the gate gets a third form: TestServeMountMetricsParity asserts every metrics
  registration the FUSE mount makes is also made by serve nfs or documented inapplicable
  with a reason. The three gates now cover the flag exists (TestServeMountFlagParity),
  the config field is set (TestServeMountBlockStoreConfigParity) and the metric is
  exported — each catching what the previous one structurally could not.

• doctor's NIC verdict claimed "no IMDS type" on a box where IMDS had returned one
  (#317). Reported from the
  c8gn.48xlarge this was built for, and it is my own regression from #394. Before that
  change, source=fallback was reached only when IMDS gave nothing, so the fixed
  parenthetical was true. #394 added a second route — IMDS names the type and the size-keyed
  estimate declines because the family is network-optimized — and I did not update the
  message for the path I was adding. The result was one report contradicting itself two rows
  apart: no IMDS type directly above a row naming c8gn.48xlarge and explaining why no
  estimate was offered for it. The #388 shape again, this time inside a single doctor run.

• doctor's fuse3 install hint named apt only — wrong on the AL2023 AMI the check
  actually fires on, which needs dnf. Also reported from the same run. Both package managers
  are now named, and a test fails if either is dropped.

• A strided slice reader fetches its extents instead of a whole block per read
  (#222). A FITS 2-D cutout walking row
  segments at a constant stride is classified Strided, and both mechanisms built to stop
  exactly that over-fetch skipped the state: the byte-exact demand lane gated on
  state == Random, and the evidence gate is consulted only on the Sequential ramp.

  The two over-fetch paths are over-determined, and either half alone makes it worse. That
  is the whole reason this was not a one-line change, and two previous attempts failed on it.
  Measured on the same fixture, steady-state bytes per 4 KiB read:

  demand lane	strided prediction	B/read
  Random only	whole block	990,321	baseline (what shipped)
  + Strided	whole block	1,001,244	+1.1 % worse
  Random only	bounded	1,044,935	+5.5 % worse
  + Strided	bounded	61,895	16.0× better

  Extending the demand lane alone leaves the prediction fetching the whole chunk, so the
  byte-exact read lands on top of it rather than inside it. Bounding the prediction alone
  changes nothing at all, because the prediction is for the block the next read demands —
  so a whole-chunk demand read asks for everything the bounded prefetch skipped and the
  per-chunk singleflight joins the two into one whole-chunk GET anyway.

  The prediction is correct about which block (at stride d it is exactly the next read's
  block, every time) and wrong only about how much, so the fix gives it a way to say so:
  prefetch.Dispatch now carries the length of the read that produced it — a fact, not a
  policy — and the FUSE layer applies the same byteExactThreshold it already uses for
  demand reads, so one rule lives in one place. BlockStore.PrefetchRange is the entry point
  that was missing; there was previously nothing between "fetch one block" and "fetch nothing".

  Cold is deliberately excluded, so the predicate is Random || Strided and not
  != Sequential: #233 establishes that a cold sequential read should fetch more, not
  less, and TestColdSequentialGetShape catches the mistake (block 0 must still cost exactly
  8 GETs). A strided bulk reader reports a whole-chunk read length, fails the threshold,
  and keeps whole blocks — its next block genuinely is wanted whole.

  Extent prefetches are labelled kind="prefetch-range" in lith_fill_bytes_total and
  lith_fill_seconds, distinct from whole, so the bytes moving between the two is what
  the fix looks like from outside. They commit only their extent span against the #313
  pressure gate, not a whole chunk per chunk — charging chunk-granular would over-report the
  quantity that gate admits against by up to 16×.

  Not fixed here, and named rather than implied: the seek re-anchor
  (prefetch.go's Sequential || Strided branch) still dispatches two whole blocks per
  landing, so a strided reader that occasionally jumps out of band still pays that path. It is
  the third of #222's three over-fetch paths and it needs its own measurement.

• --mem-cache's per-daemon arithmetic was wrong by 2×, and the refuted additive claim was
  still in the same docs row (#314). The GC
  model #392 proposed is now measured, externally, in three arms that differ 3× in
  outstanding prefetch:

  arm	peak RSS	next_gc	RSS / next_gc	next_gc / tier
  default	20.38 GB	20.30	1.00	2.03
  evidence gate forced off (pressure gate on)	20.43	20.31	1.01	2.03
  both gates off (pre-#389)	20.41	20.31	1.00	2.03

  next_gc is 2.03 × the tier every time and RSS lands within 1 % of it. The additive model
  is excluded, not merely unsupported: committed − unread_resident varied 3× across those
  arms (3.8 → 8.0 → 11.3 GB) while RSS stayed flat at 20.4 GB. RSS never exceeded next_gc,
  so it is not the allocator holding pages either.

  Two things were wrong in the docs as a result. --mem-cache is per-daemon and the
  footprint is ~2× the tier, so those compound: five mounts at the default 25 % are a
  ~250 % cap on the box, not the 125 % the page claimed. And the --mem-cache row still
  ended with the additive sentence — "the footprint is the tier plus outstanding prefetch,
  measured additive to 0.2%" — which #392 had already refuted at the start of the same
  row. Exactly the drift that #388 was about: the correction went in one place and the stale
  claim stayed in another, inside one table cell.

  The startup warning now states the measured basis rather than deriving the doubling, and
  names the one constraint on GOMEMLIMIT that follows from it: the limit must sit
  comfortably above the tier, or the collector thrashes trying to reclaim a live set it
  cannot shrink.