v1.18.2

Security

• Built with Go 1.27.3 (was 1.27.2) for
  GO-2026-6809: crypto/x509 now distrusts compromised
  intermediate certificates. lith reaches the affected verification on every HTTPS read from S3.

Changed

• Dependency updates: aws-sdk-go-v2 config 1.33.10, service/s3 1.114.5, feature/s3/manager
  1.23.16; prometheus/client_golang 1.25.0; zeebo/xxh3 1.1.0. No behaviour change is intended.

v1.18.1

Fixed

• Temporary credentials in the shared credentials file were read once and never again.
  lith signs with the AWS SDK's default chain, and the SDK's credentials cache re-retrieves only
  credentials that report an expiry. Verified against the pinned SDK (config v1.33.4): temporary
  keys with a session token, whether exported as environment variables or written to
  ~/.aws/credentials by a SAML/SSO helper, come back with CanExpire=false. So the SDK kept the
  first set for the life of the process, a helper rotating the file was never seen, and the
  mount began failing every uncached read with EIO when that first set expired — an hour in, for
  a typical SAML role session.

  lith now re-reads the credentials file every 5 minutes when the credentials in it are
  temporary, so a rotated file is picked up without remounting. The interval is a third of the
  shortest session STS issues (15 minutes), not a tuned value. Instance roles, IRSA / Pod Identity,
  SSO and assume-role profiles and credential_process already renewed themselves through the SDK
  and are unchanged.

• lith_prefetch_evidence_clamped_total's help said it was "zero unless
  --readahead-evidence-ratio is set" — stale since #368 made the ratio region-derived and on by
  default in-region. An operator on an in-region mount reading that help would conclude the
  counter could not apply to them, on exactly the mounts where it does. It now says what the
  counter answers: when it is high, the evidence gate and not the divisor bounds windows on that
  mount. Found while working out whether #312's idle handles cost anything, which is the question
  this counter decides.

• --memory-limit's docs row now carries the measurement
  (#314): the configuration that was OOM-killed
  before (a 20 GB tier on a 33 GB box) survives in both reps on v1.18.0, RSS plateauing at 28.8 GB
  against a 29.7 GB limit, with no measurable wall cost on a link-bound workload. Stated as "free
  on a streaming workload", not "free" — concurrent GC CPU was not captured. Also records that
  the limit is exported as go_gc_gomemlimit_bytes, which the row previously omitted.

Added

• A startup warning, a doctor warning, and a specific error for expired credentials. A
  session token in the environment cannot be renewed by any running process, so lith now says
  so at startup and names the sources that do renew; lith doctor grades credentials by whether
  they will survive a long mount instead of passing anything that resolves. And when S3 rejects a
  request as signed with expired credentials (ExpiredToken, ExpiredTokenException,
  TokenRefreshRequired), lith logs it at ERROR — at most once a minute, with the remedy for the
  source in use — where before it was indistinguishable from any other failure. RequestExpired
  is deliberately excluded: it is clock skew, not credentials. docs/knobs.md gains a table of
  which sources survive a long mount.

• docs/knobs.md says what to do about unfair concurrent readers on a cross-region mount
  (#391), from a cell run for the purpose:
  us-west-2 reading us-east-1, lith v1.18.0, scripts and raw data in bench/x391/. With the
  evidence gate off — the cross-region default — 16 concurrent readers against a contended tier
  came out at spread 2.70–3.18, reproducing #391; any evidence ratio from 2 to 64 brought
  it to 1.17–1.63 and finished the job 15–30% sooner, on identical bytes.

  The default is unchanged, and the reason is the instrument. The gate is off cross-region
  because one fast whole-object reader measured 1.96× slower with it on. This cell's box streams
  ~375 MB/s per reader, latency-bound rather than NIC-bound, which is the configuration least able
  to show a ramp penalty — so its single-reader null (ratio 16 at 0.94–1.00×) does not refute it.
  Ratio 64 did show a penalty, ~3× in 3 of 5 reps, so the documented range is 2–16.

  And the first run missed the regime, which is recorded rather than dropped. Its working set
  fit in the tier, nothing was contended, and gate-off came out fair at 1.27–1.32. #391's
  unfairness needs the tier to be contended; the second run used --mem-cache 1GiB to make it so.
  Cost $6.33, of which $6.02 was inter-region transfer.

v1.18.0

Added

• --memory-limit (default auto): a soft memory limit that engages only where the tier is
  already predicted not to fit (#314). On both
  mount and serve nfs.

  The tier is live Go heap, and at the default GOGC=100 the collector lets the heap grow to
  about twice the live set — measured externally at next_gc = 2.03 × the tier with RSS
  within 1% of it. So a tier over half the budget is an OOM-kill on the way to filling it.
  auto sets a limit at 90% of the budget in exactly that case and changes nothing
  otherwise: at the default 25% tier the collector's target is ~50% of the budget, far under
  any limit. Its boundary is the headroom warning's own condition (2 × tier ≥ budget), and a
  test asserts the two agree.

  This reverses #314's first-round ruling, for a stated reason. That round recommended
  GOMEMLIMIT without setting it, because the right value depends on what else is on the box.
  The objection is sound for imposing a limit on every mount and does not survive the case where
  the box is already predicted to OOM, where the comparison is not "0.90 versus 0.85" but "a limit
  under the budget versus the kernel killing the process". The 90% follows the Go team's guidance
  to leave 5–10% headroom for memory the runtime does not account for; it is not measured here.

  A GOMEMLIMIT in the environment is honoured as-is under auto; an explicit size wins over
  both. The limit in force is logged at startup, and a limit at or below --mem-cache warns,
  because the tier is live heap the collector cannot reclaim. Both commands go through one
  applyMemoryLimit, with an AST check that every command sizing a tier applies it — this
  project has seven recorded cases of mount and gateway wiring one mechanism differently.

• --demand-precise-nontiling: a demand read can be served its extents when the coverage
  signal has judged the handle scattered (#408).
  Experimental, off by default, and the reason it is off is the interesting part.

  The finding corrects the diagnosis on the issue. A ZIP read of a 10 MB archive — central
  directory, then 20 random 20 KiB entries — fetches the whole archive, confirmed externally
  against a CloudTrail ledger. The report concluded this was a demand-chunk-size floor and not
  fixable by prefetch policy. The second half is right; the first is not. lith has a byte-exact
  demand lane and these reads are far under its threshold (32 KiB against 1 MiB). What blocked it
  is the lane's predicate, Random || Strided — states denominated in blocks. A 10 MB object
  over 8 MiB blocks is 1.25 blocks, and measured, 20 of 22 reads land in the same block, 1
  adjacent, 0 further. The handle sits in Cold forever. Its trailing coverage meanwhile is
  0.056 against the 0.50 threshold with the window full: the block-space machine has nothing
  to say and the byte-space signal is emphatic.

  So prefetch.NotTiling is a third qualifying claim rather than a loosening of the first two —
  not "we have not decided" and not "the block deltas are wild", but "the bytes this handle
  touched do not tile". It is byte-denominated, so object size cannot defeat it. With it on, the
  ZIP shape fetches 3.3 MB instead of 9.4 MB.

  ★ And it is off by default because it trades bytes for round trips, with the arithmetic against
  it. The same shape's fills go 9 → 16: whole-chunk fetching amortizes faults for a
  scattered reader whose entries cluster — 20 entries across 10 chunks means two often share one
  fetch. 6.2 MB saved buys 5–20 ms of transfer; 7 extra round trips cost 210–990 ms at a
  30–141 ms first byte, a loss in every regime. That is the same bytes-for-wall shape as the
  default rolled back one release earlier, caught here before shipping rather than after, by
  reading the GET count in my own test output.

  So the right reading of #408 is not a defect but a deliberate latency-versus-bytes trade,
  which lith currently makes in favour of latency — correctly for most mounts. The flag is for the
  case where egress cost dominates: a cross-region or cross-cloud mount reading archives it
  touches sparsely.

  What it cannot reach, and the guard is pre-existing. A tiling stream has coverage ~1 and
  never matches; TestColdSequentialGetShape, the #233 witness, catches a version that drops the
  window-full requirement, so the safety property was already pinned rather than invented for
  this. And a handle with a committed window cannot match either: the coverage judgement and
  establishment are mutually exclusive by construction, which also means nothing is
  prefetching for a handle this matches, so a precise read cannot land on top of a whole-block
  prefetch — #222's measured-worse case.

• The sub-block witness now covers request count, the third axis (#407).
  An external bracketed cell on v1.17.1 confirmed the lead closes the coverage gap — uncovered
  misses back to the =false level, 62–65 against 60–62, where span-once gave 81 — and the real
  netCDF4 tool came out neutral. But the index-then-stream witness read a median 0.854, and
  the only remaining difference between arms was ~+41 GETs for identical bytes.

  Reproduced locally: 62 → 79–81 GETs on that shape with bytes unchanged. 41 extra round trips
  at a 30 ms first byte is 1.23 s against a ~7.4 s stream = 17%, which brackets the 11–25% of
  their slow reps.

  So a witness for this arm has to cover bytes, coverage and requests. It covered bytes,
  which is how a 20% regression shipped; then coverage, which is how the lead fix was verified;
  this is the third. The span dispatch count is now asserted rather than the GET count, because it
  is the deterministic driver — 16 in three consecutive runs while GETs came out 79, 81, 79.

  The cause is named and not yet fixed, with the arithmetic for why: 16 dispatches is about 5×
  what this shape needs. The span phase covers only the first ~8 MB of the stream, and across it
  the lead grows from ~4 MB to ~16 MB, so the frontier runs away from the reader and ~3
  dispatches would cover it. The extra ones happen because the frontier is recomputed as
  readEnd + earned on every read and earned grows faster than the reader advances, so the
  comparison against the previous frontier never short-circuits. Fixing it needs a trigger tied to
  the remaining lead rather than to the earned amount — a tuning decision better made from a
  measurement than invented, which is why this release asserts the current count instead.

Fixed

• Inside a container, lith sized its memory against the host's RAM
  (#314). /proc/meminfo is not namespaced by
  the memory controller, so a container reads the host's MemTotal. A container limited to
  4 GiB on a 256 GiB host therefore defaulted to a 64 GiB memory tier — sixteen times its
  whole allowance — and the startup headroom warning, comparing against the same figure, stayed
  silent. The image's entrypoint is lith serve nfs and a Kubernetes pod with a memory limit is
  the ordinary deployment, so the shipped default was an OOM-kill on the ordinary case.

  The memory budget is now the smaller of MemTotal and the cgroup limit (v2 memory.max,
  v1 memory.limit_in_bytes), with both spellings of "unlimited" recognised rather than parsed —
  v2's literal max and v1's page-rounded near-MaxInt64. The default tier, the headroom warning
  and the new soft limit all read it.

• TestStreamingByteIdentical's fixture was not a stream. It read 64 KiB at every 1 MiB
  boundary — 1/16 density — and asserted that a "strictly sequential read run" never goes
  byte-exact. Measured, that handle never leaves Cold, never establishes, has window 0 and
  prefetches nothing, and its trailing coverage is 0.0625. The name and the premise described a
  reader the fixture did not contain, and the chunk-alignment assertion was protecting a property
  that reader does not have.

  Found by #408: extending the demand lane took that reader from 16× to 2.4× and the test
  failed — correctly reporting a change, incorrectly calling it a leak. The fixture now reads
  whole chunks, which is what its intent requires, with preconditions asserting the handle
  actually establishes and is judged tiling. The 1/16 case is now its own test, recording that it
  is served precisely and why that is safe there.

v1.17.1

Fixed

• A sub-block span now keeps a lead instead of being placed once
  (#407). This is the cause of v1.16.0's rolled-back
  regression, fixed; the flag stays off by default.

  The span bounded the dispatch a handle got at establishment and was never extended. The
  whole-block frontier cannot move again until the reader crosses a block — 64 reads at the
  default 8 MiB block and a 128 KiB read — so a handle that went on streaming faulted through
  everything the one span did not cover. Measured externally at median 0.798 throughput with
  bytes identical: the entire cost was faults and wall.

  The span now holds the frontier earned bytes ahead of the reader and is extended as the reader
  consumes it, including within a block, which is where the detector previously returned
  early for every read that did not cross a boundary. That is the evidence-gate rule expressed in
  bytes rather than blocks: a handle that keeps reading contiguously earns a deepening lead and
  converges on what the whole-block path would have given it, while one that stops has committed
  only what it had read.

  Dispatch now carries an absolute span range (SpanOff, SpanLen) rather than a length the
  FUSE layer placed at the read's end. An extension begins where the previous span ended, which
  only the detector knows — the original split was elegant and could not express the fix.

  The cost is some of the byte saving, which is the right trade and is stated rather than
  elided: a 64-field walk fixture goes from 12.1× to 8.6× less dispatch, and the end-to-end
  figure from 2.03× to 1.87× fewer bytes.

Added

• A deterministic guard for the axis the previous witness missed. The earlier
  scatter-then-stream test asserted bytes and span count; bytes were identical and the whole cost
  was faults and wall, so it passed and the regression shipped. Fault counters are too noisy in a
  fake-server fixture to assert on, but dispatch coverage in byte space is deterministic — it
  is a function of the decisions, not of timing — so that is what the new test asserts.

  It sweeps the stream's phase relative to the block grid, which is not decoration. At a
  block-aligned start the handle establishes 64 reads in with a near-floor span and the gap does
  not appear at all; a first version of the probe used an aligned base and reported the two arms
  identical, which reads exactly like "no regression". The gap is largest when the handle
  establishes early, having earned little:

  stream phase	established at read	uncovered, whole-block	uncovered, span placed once
  +0 MiB	64	65	65 (invisible)
  +2 MiB	48	49	73
  +6 MiB	16	17	105
  +7 MiB	72	73	73 (invisible)

  With the lead maintained, every phase matches the whole-block arm. Three further properties are
  pinned because a revert sweep found each one ungated: dispatches stay a small fraction of reads
  and never re-cover ground; a walk read backward gets as many spans as the same walk read
  forward (the span frontier is absolute, so failing to reset it at a run boundary silently
  starves every subsequent run — 35 spans became 7); and the FUSE layer passes the dispatch's own
  offset, checked by an AST walk because no behavioural test in that package can see it.

  One honest note recorded in the test: the frontier's early-return is defensive and not
  exercised by any measured shape. Removing it left the dispatch count at 48 of 512, unchanged,
  because earned grows twice as fast as the reader advances. The dispatch-count assertion guards
  the floor check, which is load-bearing, rather than the frontier.

v1.17.0

Changed

• --readahead-subblock-dispatch is off by default again, reverting v1.16.0
  (#407). It costs wall clock on an
  index-then-stream reader, which v1.16.0's evidence did not cover.

  shape	bytes	wall, median B/D	uncovered misses
  GRIB .idx sweep	3.02× → 1.39×	1.005 (11 kept reps)	69 every arm
  HEMCO time-slice	1.79× → 1.48×	0.974 (5 kept reps)	331–333 every arm
  index-then-stream	identical	0.798 (5 kept reps)	62 → 81

  The last row was below both of its brackets in 8 of 8 reps. The mechanism is the first
  dispatch at establishment — one ~5.5 MB span in place of three whole blocks (24 MiB) — after
  which a streaming reader faults through the uncovered remainder until the next block crossing;
  the window schedule after that is identical in both arms.

  The trade is bytes on a walk that stops against wall on a walk that continues, and which
  of those a mount is cannot be known at establishment. That makes it an opt-in, not a default.
  The 13-shape sweep was otherwise clean and the default did wire through, so nothing else is
  affected either way; webdataset_stream was neutral over 8 kept reps, confirming the default
  cost nothing where it did nothing.

  Turning it back on wants a span that still leads the reader rather than only shrinking the
  first dispatch — not a re-run of the same cells.

Fixed

• The witness built for this exact risk asserted the wrong axis. internal/fuse's
  scatter-then-stream test was written before the arm shipped, specifically to catch a handle that
  scatters and then streams, and it asserts bytes and span count. Bytes are identical on that
  shape; the entire cost is in faults and wall. It passed, the default went on, and the regression
  shipped.

  And it cannot be fixed by adding an assertion there. The fault counts are now logged, and
  they are too noisy in that fixture to gate on: across three identical runs the arm's extra
  faults came out +5/+1/+0 on the short stream and +4/+4/+11 on the long one, so even a contrast
  between the two already fails. UncoveredMiss fires when a demand read finds its chunk neither
  cached nor in flight, so it races the prefetch goroutine, and a fake server has no latency for
  that race to resolve against. The external cell needed 1 GiB streams and 8 bracketed reps to see
  62 → 81 cleanly.

  So the figures are logged with a comment that says plainly a green run of that test does not
  clear this arm — the wall and fault axes are external-only. Writing a flaky assertion there
  would have been worse than writing none.

v1.16.0

Warning

Superseded by v1.17.0 — upgrade.

This release shipped --readahead-subblock-dispatch on by default, and that default is
rolled back in v1.17.0. It costs wall clock on an index-then-stream reader — a handle that
reads scattered and then streams a large contiguous range, which is the common
metadata-then-payload shape. Measured after this release: median 0.798 throughput against
the flag off, below both brackets in 8 of 8 reps, with uncovered demand misses 62 → 81 and
bytes identical.

The flag itself is unchanged and still available. If you know your workload reads slices and
stops, --readahead-subblock-dispatch still gives the byte savings described below; the
correction is only that it is the wrong thing to turn on for everyone. See
#407 for the measurement.

---

Changed

• --readahead-subblock-dispatch is on by default (#407).
  Pass =false to disable. This was pre-registered: wall clock neutral on both moving rows with
  faults flat meant the default moves on.

  	GRIB	HEMCO
  byte amplification	3.02× → 1.39×	1.79× → 1.48×
  wall, median B/D	1.005 (11 kept reps)	0.974 (5 kept reps)
  uncovered demand misses	69 in every arm	331–333 in every arm
  GETs	71–74	339–341

  Each flagged run was bracketed by two default runs, with a rep discarded when the brackets
  disagreed by more than 10% — necessary because GRIB's ~3 s wall drifts 15–24% between brackets,
  which is larger than the effect. The tester disclosed that the flagged arm looked faster in
  the first six reps and that the impression did not survive the rule: the high-drift reps were
  exactly the ones it discards.

  FITS, three control shapes, a whole-object dd and a scatter-then-stream handle are all
  unchanged, and the arm cannot fire where the evidence gate is off, so cross-region and
  region-unknown mounts are unaffected. The default reaches exactly the in-region regime the
  measurement covers.

Added

• The sub-block arm's test now reports uncovered demand misses, which is the mechanism by
  which it could cost wall clock (#407). Bytes
  a reader never wanted are pure waste; bytes it did want and now has to fault for are a stall,
  and a byte count cannot tell those apart — which is the whole reason the default decision is
  open after a clean byte result.

  On the walk fixture the arm halves bytes and triples uncovered misses (4 → 12) while
  prefetch hits fall 6 → 1. Treated as a bytes-for-faults trade, break-even is around
  100 MB/s of effective bandwidth at a 30 ms first byte, and ~30 MB/s at 100 ms — so if that
  fault increase transferred to a real workload, the arm would be a wall loss at any bandwidth
  those instances actually reach. An external run measured the GET count flat (72 → 71) on the
  real shape, which argues it does not transfer. The two signals disagree, and only a
  wall-clock cell on the real workload settles it.

  Logged, not asserted. Any bound on this ratio would be a tuned constant on a
  fixture-specific quantity, which is how both #340's and #349's withdrawn defaults were placed.
  The figures are in the test output so a future change to this arm shows its effect on faults
  and not only on bytes.

Fixed

• --readahead-subblock-dispatch's help and docs row carried fixture figures and a stale
  "measured inert" note (#407). Both now carry
  the external v1.15.0 measurement, two arms and two reps with the default arm reproducing the
  previous release exactly as the control:

  shape	default	with the flag	GETs	prefetch-range
  GRIB .idx sweep	3.02×	1.39×	72 → 71	0 → 5.05 MB
  HEMCO time-slice	1.79×	1.48×	340 → 339	0.26 → 4.98 MB
  FITS cutout	2.20×	2.20×	12 → 12	0

  All of GRIB's whole fills become prefetch-range and its used:issued goes 2:24 → 2:5, so the
  change is attributable rather than inferred. The GET count does not rise, which settles the
  #124 concern that a byte win would arrive as a
  GET explosion: the span replaces whole-block fetches rather than fanning out.

  Two shapes qualify, not one, and the docs now say so. I predicted only GRIB would move, on
  the premise that HEMCO handles never enter Random; a trace I had already quoted showed one of
  them at 129 consecutive random→random transitions. The trigger is a property of the access
  pattern, not of a file format — any handle that reads scattered and then reads a short
  contiguous stretch meets it. Recorded so the next reader does not inherit the wrong premise.

  And the pre-registration it broke was badly written. Its failure condition — "if FITS or
  HEMCO move at all, the mechanism is not as described" — conflated whether the trigger behaves
  as specified with which shapes meet it. Only the first is a claim about the code. A falsifier
  that depends on a guess about someone else's workload tests the guess.

  Still off by default, and the row now says what is missing: wall clock on the two rows that
  move. The arm bounds prefetch, so its risk is latency rather than bytes — fetching 0.46× the
  bytes could be faster, neutral or slower, and a byte count cannot separate those. #309's 11×
  regression came from a prefetch-depth change that looked clean on totals.

• My bytes-for-faults model was a fixture artifact, and it is now pinned as one. A fixture at
  8 MiB field spacing showed the arm halving bytes while tripling uncovered demand misses,
  from which I published a break-even implying the arm would be a wall loss at any realistic
  bandwidth. The external cell then measured faults flat on both real shapes.

  The cause is the spacing: the default dispatches window + 1 blocks, which at that spacing
  overshoots into the next field, so the default's over-fetch was accidentally prefetching it
  and the arm's shorter span was not. At wider spacings the fault counts are identical and the
  byte saving is larger — and a scattered walk jumps farther than the window by definition. A
  test now measures both sides of that boundary, so the artifact cannot return disguised as a
  regression.

  It asserts the delta between arms across two spacings, not an absolute count: UncoveredMiss
  fires when a demand read finds its chunk neither cached nor in flight, so it races the prefetch
  goroutine and picks up extra misses under load. The first version asserted equality, passed
  alone, and failed twice in three package runs — the same mistake as the #313 gate test that
  asserted the gate halves peak pressure.

v1.15.0

Changed

• --readahead-subblock-dispatch now engages for a handle with a Random history, and reads
  the current run at that site (#407). Still
  experimental, still off by default.

  The arm shipped in v1.14.0 requiring a finished established run, and an external 2×2 then
  found it inert on every measured shape. The per-handle trace said why: a GRIB handle
  establishes once, in its last 12 of 90 reads, and finishes no runs at all, while 67 of
  its 90 reads are random→random. The finished-run trigger described a fixture, not the
  tools. A handle that has collapsed to Random at least once now qualifies too.

  The basis at this site is always the current run, not --readahead-evidence-per-run's
  choice, because at such a handle's establishment lifetime consumption is evidence about
  scattered reading rather than contiguous appetite — GRIB's 11.8 MB lifetime earns 23.6 MB,
  above the floor, so a lifetime basis declines on exactly the shape the arm exists for. This
  collapses the former 2×2 to one flag: the arm now works alone (measured 3.82× less dispatch
  on an 8-field walk fixture, 12.1× at 64 fields) and is independent of the per-run flag, which
  a test asserts as equality so the two cannot be quietly re-coupled.

  Why a span is the only lever that can reach this shape. GRIB's 3.02× is one
  establishment dispatching window + 1 blocks — the +1 being
  #233's rule that a handle prefetches the block
  it is in — so 24 MiB for 2 of 24 chunks used, with the window already at its 2-block floor
  at every ratio (11.8 MB consumed earns 1.41 blocks at ratio 1 and 2.81 at ratio 2; both land
  on 2). Neither a ratio change nor per-run evidence can move that, which an external ratio-1
  null independently confirms.

  The risk was measured, not argued. The shape that makes a Random-history trigger
  dangerous is a handle that scatters and then streams — an index read followed by the payload
  — since the trigger fires exactly when it starts wanting a lot. A witness fixture written
  before the change measures the cost to such a stream at zero, and non-scaling: one span at
  establishment whether the stream is 16 MiB or 48 MiB, with the window ramping back. Part of the
  protection is arithmetic rather than policy: establishment requires a block advance, so a dense
  stream starting at a block boundary has consumed a full block by then, earns exactly the floor
  at ratio 2, and the span declines on its own.

  A plain sequential copy satisfies neither route — the #221 coverage gate holds its first
  reads provisional as Cold and it never passes through Random — which is what keeps #233's
  cold streaming reader out, and is now asserted on both counters rather than one.

Added

• docs/knobs.md prices what stopping early costs, which turns out to be the yardstick the
  last three over-fetch reports were missing. A plain sequential reader that simply stops pays
  ≈ 1 + ratio — measured 2.94–3.29× at ratio 2 and 2.00–2.35× at ratio 1, stable across
  stop points, against 3.8–15.1× with the gate off. That is the designed trade: the lead is what
  makes a stream fast and a reader that abandons it pays for it.

  It reframes three rows that had been grouped as one family for weeks. Two are at or below
  the line and are not defects (a FITS cutout at 2.20×, a HEMCO time-slice read at 1.79×); only
  the GRIB .idx sweep at 3.02× exceeds it, and for the specific floor-bound reason above. A
  shape fetching ~3× at ratio 2 is paying the price of stopping early; only one fetching
  materially more has something else wrong.

Fixed

• --readahead-subblock-dispatch's help and docs row claimed a benefit no measured shape
  shows (#407). An external 2×2 on the v1.14.0
  artifact found all four arms identical on a GRIB .idx sweep (3.02×), a FITS cutout
  (2.20×) and a HEMCO time-slice read (1.79×) — with zero prefetch-range fills attributable
  to the arm, GETs unchanged, and a clean dd witness. The flags were verified present on the
  live process before the null was reported.

  The precondition is why. The arm requires a finished established run, and the real tools
  finish roughly none: a GRIB handle establishes once, in its last 12 of 90 reads, and
  finishes none — 67 of its 90 reads are random→random; a FITS handle is a single 130-read
  run that ends only at close and never enters Random. The synthetic fixture the arm's 12.1×
  comes from re-establishes once per field, which is not how these tools read; at a regular
  field spacing the same walk classifies as Strided, a third branch again.

  The span machinery is kept and the trigger is what is refuted — bounding a dispatch in
  bytes, fetching it through the extent path and labelling it prefetch-range are all
  fixture-proven. Both texts now say the arm is measured inert and why, rather than quoting the
  fixture's figure as if it were a shape anyone reads. Second time this issue has needed that
  correction, which is the #388 drift.

  Also records what GRIB's 3.02× actually is, now that the external per-handle trace makes it
  attributable: one establishment dispatching window + 1 blocks — the +1 being #233's rule
  that a handle prefetches the block it is in — so 24 MiB for 2 of 24 chunks used, with
  windowCap already sitting at its 2-block floor. That is a single over-wide dispatch, not an
  accumulation of bounded short runs, so a mechanism aimed at repeated runs cannot reach it.

• Nothing asserted that a finished run was ever counted. Found by a revert sweep: once the
  Random-history route existed, reverting the deEstablish/endRun ordering bug stopped
  failing anything, because the arm still fired through the other route. runsEnded could have
  been permanently zero — a dead route behind a green suite, the ttfbRecorder shape. The new
  test runs through the FUSE layer deliberately: at the prefetch level the coverage gate cannot
  be disabled, and with it off the jumps take the seek re-anchor and call endRun directly
  with established still true, so the ordering bug is invisible there.

v1.14.0

Added

• --readahead-subblock-dispatch: a readahead dispatch can be shorter than one block
  (#407, arm B). Experimental, off by default.

  The evidence gate is denominated in blocks and floors at 2, so the smallest window any
  established handle can be given is 16 MiB at the default 8 MiB block — while a GRIB .idx
  field sweep, an HDF5 chunk walk and a FITS cutout's row bands all read 0.5–2 MB per run.
  This converts that residual from blocks into bytes: a handle that has earned less than the
  floor is dispatched a contiguous span of exactly what it earned, placed immediately after the
  read that earned it, and fetched through the same extent path a strided prediction uses.

  This is what makes --readahead-evidence-per-run able to do anything. That arm shipped in
  v1.13.0 and an external sweep then measured it identical to the default on all three walk
  shapes. The null was structural — arm A correctly shrinks the earned window and the block floor
  hands the same dispatch straight back. Measured locally on a 7-field walk fixture, dispatched
  bytes over the 2×2:

  	arm B off	arm B on
  per-run off	baseline	decays to 1.03×
  per-run on	1.00× (exactly inert)	12.1×

  Neither arm alone holds up, and they fail for different reasons — which is why they are two
  flags rather than one default. Arm A is exactly inert, to the byte. Arm B alone is not inert
  but decays: it reads lifetime consumption, which grows past the floor's worth within a few
  runs, after which it switches itself off — its span count pins at 13 whether the walk is 8
  fields or 64, while the pair issues one per landing indefinitely. This is #222's
  precedent repeating, where each half alone made bytes worse and only both together gave 16×.

  End to end against a fake S3, the pair takes a walk from 7.20× to 3.55× over-fetch —
  2.03× fewer bytes — moving fills from kind="whole" to kind="prefetch-range", which is
  how the change is visible from outside.

  It cannot reach a streaming reader, and that is a precondition rather than a threshold. The
  arm engages only for a handle that has finished an established run, which a sequential copy
  never does. Today the 2-block floor is what protects that reader, so an arm that removes the
  floor has to carry the protection itself — #233
  says a cold sequential read should fetch more, and #231
  refuted three attempts to make it fetch sooner.

Fixed

• --readahead-evidence-per-run's help motivated it with figures from a ratio that is no
  longer the default (#407). The flag's help
  and its knobs.md row quoted "48 chunks prefetched and 2 used on GRIB ... a peak window of
  5 blocks", all measured at ratio 4 — and v1.13.0 moved the in-region default to 2, at
  which point the flag does nothing at all on that shape. 11.9 MB at ratio 2 over an 8 MiB
  block earns 2.84 blocks, which floors to the same 2 blocks a 0.5–2 MB run earns, so both
  bases return the identical window.

  Caught by an external null result that was structural rather than informative. A sweep on
  the v1.13.0 artifact measured this flag identical to the default on all three walk shapes and
  recommended retiring it. The arms were identical by arithmetic on that shape, so the cell
  had no power to separate them — the same reading that made ratio 1's tie with ratio 2
  uninformative, pointed the other way. The flag is kept.

  Discrimination starts at a lifetime consumption of 3 × block ÷ ratio = 12.6 MB at the
  defaults, and the measured shape sits 5.2% below it. Above the boundary the cumulative
  basis grows without bound — 4 blocks at 20 MB, 23 at 100 MB — while a short run stays pinned
  at 2. Both the help and the row now state the boundary, and a test pins it, because the only
  test of this arm's mechanism ran at ratio 4 and so kept passing while the default moved out
  from under it.

• A finished run was not counted, so arm B was inert on every shape. deEstablish cleared
  established before calling endRun, and endRun reads that flag to decide whether there
  was a run to end at all — so it always saw false, no run was ever counted, and the
  precondition above could never be met. Found by instrumenting a walk fixture that
  re-established four times and reported zero finished runs; every decision-level test for the
  arm was green while it could not fire end to end, which is the ttfbRecorder shape again.

• The two-block dispatch floor was documented as 17 MiB; it is 16 MiB. 2 × 8 MiB is 16 MiB
  (≈17 MB), and the wrong unit had propagated into --readahead-evidence-per-run's help,
  docs/knobs.md and two places in v1.13.0's changelog section. Corrected in all four,
  including the released section — a wrong figure where people read it is worse than an
  edited historical entry — so that section now differs from the v1.13.0 release body as
  published. Same class as #340's 5 ms, which was the wrong unit rather than the wrong value.

v1.13.0

Added

• --readahead-evidence-per-run: the evidence gate can measure the current run instead of
  the handle's lifetime (#407). Experimental,
  off by default. The first arm of a 2×2 that has not been measured yet.

  --readahead-evidence-ratio bounds a committed window by the bytes a handle has read — but
  that figure is cumulative and never reset. On a scattered-but-locally-sequential walk
  every run is 0.5–2 MB, so by the third run the handle has earned a window no single run can
  justify, and it re-places that bet at every re-establishment. An external shape sweep measured
  the consequence across three shapes: 48 chunks prefetched and 2 used on a GRIB .idx
  field sweep, 71/11 on a HEMCO time-slice read, 47/13 on a FITS cutout — one mechanism, three
  findings (#407, #409, #222).

  The arithmetic reproduces their measurement exactly: 11.9 MB consumed × 4 ÷ 8 MiB = 5.69 → 5
  blocks, against a measured peak window of 5. A test asserts that correspondence, so if
  either the arithmetic or the measurement moves, the mechanism has to be re-derived rather than
  assumed.

  A run ends at a seek out of an established pattern, a scattered landing, or a
  de-establishment. A genuine sequential copy never reaches any of those, so its run is the
  whole file and this is a no-op for it — that is the witness three cold-start fixes were
  refuted against (#231), and the reason the reset sits at run boundaries rather than on a timer
  or a read count.

  Necessary and not sufficient, which is why it is a flag and not a fix. The dispatch floor
  is 2 blocks — 17 MiB at the default block size — so a 1 MB run still floors far above what
  it wants. Closing the rest needs sub-block dispatch, which is not built. The one precedent
  for this pair (#222) found each half alone made bytes worse and only both together gave 16×,
  so the halves have to be measurable separately.

• A guard that experimental flags default off. Found by a revert sweep: flipping this flag's
  default to true left the entire suite green. "EXPERIMENTAL" in a flag's help is a promise
  that the behaviour is unmeasured, and a binary that ships it on contradicts its own help where
  no operator can see it. Keyed on the help text rather than a list, so a new experimental flag
  is covered the moment it is written.

• lith_streaming_handles_idle_{1s,5s,30s}: the idle distribution over the readahead
  divisor's own population (#312). An
  instrument, not a mechanism — nothing reads these.

  Because the measurement I asked for could not answer the question. I asked for
  lith_streaming_handles against lith_open_handles, and it came back 49 against 160. That
  says #311's divisor input is working — it excludes 111 descriptors, 69 % of them — and says
  nothing about idleness, because a stream is counted whether it read a microsecond ago or
  a minute ago. Both pre-registered rows on that issue were really about #311.

  A follow-up timeline cell established that the idleness is real: on the dominant mount, 7 of
  12 keys had gaps ≥ 2 s carrying 91 % of demand events, median long gap 5.0 s, and
  72 % of those gaps contained no re-open of the key — so a held descriptor sat through
  them. What it could not establish is magnitude, because the timeline is per key and
  cannot see reads the kernel serves from the shared page cache. These gauges are per
  handle, and see exactly the reads the divisor's input is computed from.

  A distribution rather than a verdict, for a stated reason: #312's mechanism needs an idle
  threshold, and that threshold is a tuning constant with no principled value — too short and a
  reader pausing on compute loses its window, too long and the mechanism does nothing.
  Exporting the shape lets the next measurement choose it. The buckets themselves are
  measured, not chosen: 1 s sits inside a read burst, 5 s straddles the ExtData time-slice
  cadence, 30 s is past any slice.

  One walk of the handle map per scrape, through a single collector rather than four
  GaugeFuncs — with four, one scrape could observe three buckets at one instant and the
  denominator at another, and a stream going idle in between would make the denominator read
  smaller than a bucket it contains.

• Docs: a root-created mount needs a root-run readiness check. Reported from a benchmark
  harness: mountpoint -q as a non-root user cannot stat a root-owned FUSE mount, so it
  reports not mounted while the mount is in fact serving — a readiness loop times out and, in
  their case, three mounts stacked up behind one failed check. lith mounts does not rescue
  you either, for a different reason: it reads a per-user record directory and deliberately
  ignores records from a directory it does not own, so a root-created mount is equally
  invisible to a non-root lith mounts. The rule is the same for both tools, and it was in
  neither the docs nor anyone's head.

  Both directions, because the note cost a measurement in the other one. The first version
  of this warned only about a non-root check against a root-owned mount. A later harness ran
  sudo mountpoint against a user-owned mount, which fails the same way — every cell in
  that sweep sat idle for its full 60 s timeout, and the run initially read as a 1.17×
  speed-up that the reporter then withdrew. The note now says the check and the mount must be
  run by the same user, rather than naming one of the two mismatches.

• lith_fill_seconds{kind}: what a fault costs, as a scrape rather than a judgement
  (#362, condition 2). An operator mounted a
  1.206 TB kraken2 database, ran a tool that classified zero reads in eleven minutes, and
  spent three instances and most of a day establishing that the mount was working
  correctly — it was in #232's regime, a
  serial fault stream at ~141 ms per 4 KiB probe, which lith has no lever to speed up. In
  their words: "the per-fault latency, the fill kind, and the random-vs-sequential
  classification are all things lith knows and I couldn't see."

  Fill first-byte latency is now exported under the same kind label as
  lith_fill_bytes_total. kind="demand" is a fill no prediction covered, which is the
  fault-stream shape: its quantiles answer what is a fault costing me, its _count answers
  how many faults, and its share against kind="whole" answers is this mount streaming or
  faulting. Those are the three things they could not see.

  Labelled by fill kind rather than by detector state, which is what the issue's framing
  suggested. The kind is already decided at every call site of the fetch seam, where the
  prefetch detector's state lives in internal/fuse and would have to be plumbed down through
  the blockstore; and the kind is the more direct answer, because it says what lith did
  rather than what it predicted.

  No threshold and no verdict. Read it with lith_ttfb_floor_seconds, which is
  load-invariant by construction, and the diagnosis reads itself: demand fills at ~141 ms
  against a ~19 ms endpoint floor is one round trip per miss. docs/copy-or-mount.md now
  carries the three queries and how to read them, including the case that is not this
  regime — demand latency far above the floor is queueing, not S3. Two defaults placed from a
  plausible-looking constant have already had to be withdrawn here
  (#340,
  #349), and "is my workload in this regime"
  is a question about the workload.

Changed

• The in-region evidence-gate ratio is 2, not 4 (#407).
  --readahead-evidence-ratio's region-derived default now bounds a committed window to twice
  the bytes a handle has actually read. Over-fetch on the shape the ratio was originally set
  for — #284's single-variable NetCDF-4 read — goes 2.66× → 2.28×; nothing measured got
  worse.

  This is a smaller default, not a new mechanism, and it was decided by a sweep rather than
  by argument. An external 2-vs-4 run across nine shapes found ratio 2 free or better on every
  one: the single var1 reader 59.8 → 51.4 MB, six concurrent readers 358.6 → 308.3 MB
  (2.62× → 2.26×), 0.86× bytes in both repetitions, with ratio 4 reproducing the historical
  figures to 0.1 MB as the control. netcdf4_hyperslab, webdataset, concurrent_handles and
  a single-stream dd came out byte-identical at either value.

  The ratio does not interact with the per-handle divisor. The six-reader cell stayed exactly
  6× the single-reader cell at both ratios, which is the property that makes this one constant
  rather than a constant per concurrency level.

  Ratio 1 is not better than 2, measured identical on all three walk shapes (GRIB, HEMCO,
  FITS): by then every window is pinned at the 2-block dispatch floor — 17 MiB at the default
  block size — and the ratio has stopped being the binding constraint. That residual is
  sub-block dispatch's to close, not this default's. The test records 4, 2 and 1 as the three
  values anyone has measured and fails on any other, so the next person to move it has to
  bring a cell rather than an intuition.

Fixed

• The NFS gateway's readahead divisor counted MOUNTED clients, not reading ones
  (#337). A client that sent one MOUNT held a
  share of the prefetch budget for --client-idle — five minutes — whether or not it ever read
  a byte. Measured locally with no client reading anything: 1 mount → 96 blocks, 8 → 61,
  48 → 10, 246 → the 2-block floor. **A 48× colla...

v1.12.0

Added

• The mount and the gateway warn when --mem-cache cannot fit on the box
  (#314). A 20 GB tier on a 33 GB box was
  accepted silently and the daemon was OOM-killed at RSS 30.56 GB. The check runs before
  anything is fetched, because the failure it predicts is the process dying.

  The mechanism is not the one the issue was filed with, and the correction matters because
  two of its three proposed fixes were built on it. #314 modelled the footprint as the tier
  plus outstanding prefetch commitment (20 + 13.078 GB, summing to MemTotal almost
  exactly). That addition does not hold. lith_prefetch_committed_bytes is charged at
  dispatch and released only on consume/evict/fail, so it keeps counting a chunk after that
  chunk has landed in the tier — it double-counts the resident half — and a dispatched
  chunk that has not yet been admitted holds no chunk buffer at all, because cloneChunk runs
  after fetchReader returns and after the bytes-in-flight budget is acquired. The field's
  own comment in blockstore.go says it: "it is NOT a memory figure".

  What does explain it is the collector. The tier is live Go heap (every chunk is a
  make([]byte, cl)), lith sets neither GOGC nor GOMEMLIMIT, and at the default
  GOGC=100 the heap target is about twice the live set. A 20 GB tier therefore targets
  ~40 GB on a 33 GB box and the process dies on the way there — which also explains why the
  kill came at 30.56 GB rather than at the 33.08 GB the additive model predicted.

  So the ceiling is ~50 % of RAM, and it is arithmetic from GOGC's definition rather
  than a constant anyone picked — the distinction that cost three releases on
  #340/#349.
  2 × tier is a lower bound on the target (the tier is not the only live heap), so the
  check under-warns rather than over-warns: a warning that fires on a configuration which
  would have survived teaches operators to ignore it. The message names the ceiling and
  GOMEMLIMIT, which makes the collector work harder instead of letting the kernel kill the
  process.

  A warning and not a clamp or a refusal, consistent with --prefetch-pressure-max: an
  operator who typed a number keeps it, and a tier fills lazily, so a short-lived mount over a
  small dataset may never reach the bound.

  Also documents why the default is 25 % — previously unwritten, and now load-bearing.

• A warning when --prefetch-budget and --prefetch-pressure-max compose into a stall
  (#313). Found while deriving the default
  above, by writing out the denominator rather than the interesting term.

  The gate is sized against the tier, while the quantity it limits is independently bounded
  by --prefetch-budget, which defaults to half the tier. That is the entire reason the
  new default is safe: one handle's steady-state pressure is ~0.50 against a 0.85 threshold, so
  the gate is silent until a concurrent start makes the divisor lag. Raise --prefetch-budget
  to or above 0.85 × --mem-cache and a single handle at full budget sits at the threshold,
  so the gate holds dispatches continuously and the mount throttles itself with nothing wrong —
  and lith_prefetch_pressure_held_total would climb and look like the gate working. Two knobs
  that are individually reasonable, composing into a self-inflicted stall that nothing would
  have reported.

  The mount and the gateway now warn, naming the figure to lower. BlockStore.MemCap() is
  exported so the check uses the tier's realized capacity (shards × per-shard) rather than
  the requested --mem-cache.

• --prefetch-sibling-coverage: concurrent readers of one object can establish again
  (#316). Experimental, off by default.

  lith opens with FOPEN_KEEP_CACHE, so every descriptor on an inode shares the kernel page
  cache: a sibling's reads are served by the kernel and never reach lith, each handle's own
  stream is punctate with coverage ≈ 1/N, and from four concurrent readers of one object
  the #221 coverage gate holds every handle
  provisional so nothing prefetches. Measured 243× slower than one reader alone with
  byte amplification of 1.001 — the cleanest byte count of any cell and the slowest run,
  so no byte or request counter can see it.

  A hole now counts as covered when another descriptor is open on the same object and the
  bytes in it were already demanded through lith. Both conditions are load-bearing, and the
  first was added after a test caught the fix being wrong: lith fetches a whole 1 MiB chunk to
  serve a 128 KiB read, so for holes smaller than a chunk a lone strided reader's own
  demand fetches make its own holes look demanded — which is
  #222's shape and exactly what must not
  establish. At chunk granularity the two cases are indistinguishable; what differs is who
  demanded the bytes, and "someone else has this file open" is the cheapest sound proxy.

  The demanded condition holds the band between a chunk and a block, which nothing else
  covers. Holes wider than --block-size never reach either condition — the byte-gap gate
  forces Random first, which is also why a 16 MiB-stride fixture tests nothing here and the
  first version of that test was vacuous for passing on the wrong gate.

  Confirmed externally at the default --prefetch-coverage-min: 16 of 16 readers
  establish in both reps, wall 67 s → 2.9 s, coverage_held from ~1000 to 16 (the
  same as O_DIRECT), with the holes themselves unchanged — it discounts them rather than
  removing them. It is also the most byte-efficient of the three ways to get there: 0.67 GB
  fetched for 0.54 GB read, against 1.89 GB for --prefetch-coverage-min 0.05 and
  0.86–1.53 GB for O_DIRECT.

  "Already demanded" rather than "resident" is deliberate twice over: a random walk's holes
  are not resident at all (#232 — no lever
  there, and prefetching for one adds bytes without moving the wall clock), and lith's own
  unread over-fetch is resident but undemanded, which must not count or a strided reader
  would establish on regions its own window swept.

• Docs: the per-connection cold first-traffic cost, and the evidence gate's second job
  (#381). A fourth rule in CONTRIBUTING's
  measurement section, and the product consequence in knobs.md.

  A new connection's S3 first-byte time starts at ~100–150 ms and settles to ~30 ms over
  roughly its first second of traffic — per-connection and ramp-like, not a setup cost: one
  request per connection does not fix it, and about a second of traffic per connection does.
  A client that opens many connections at once and measures immediately eats it in full: 22%
  of first bytes under 50 ms against 92% after a one-second warm-up, so a cold first arm
  reads 3–10× worse than steady state.

  The evidence gate's ramp turns out to protect against this, which nobody designed it for.
  Measured on two mounts started at the same instant: with the gate off lith opens 83% of its
  fills on new connections every rep and comes in up to 49 points colder, with identical
  bytes fetched. The same cell explains the reused-connection queueing residual from #350 —
  gate-off has zero (0.01 ms every rep) against 0.76–2.26 ms gate-on — so the ramp both
  protects against the cold cost and causes the queueing. One mechanism, two signs.

  Recorded with the consequence and the tension: gate-off mounts (cross-region, or
  region-unknown) open their connections at once and pay the cold cost on every fresh mount,
  which trades against the measured 1.96× ramp penalty at distance. That net is unmeasured,
  and the docs say so rather than implying a recommendation.

• lith-s3bench -warmup-workers: size the warm-up independently of the measured burst
  (#381). The output line now reports
  warmup=<dur>/<n>w, so an arm that warmed at a different scale than it measured says so.

  This is the variable the remaining #381 asymmetry turns on. A fresh lith mount is fast
  while a fresh lith-s3bench process at the same instant is slow — 17 of 18 simultaneous
  pairs — and the structural difference is that a mount does region resolution and an index
  read before its burst. The ladder established that a tiny separate request does not warm
  the effect (head-object and one worker for 0.2 s both left it present), so the open
  question is how much traffic does, and answering it needs the warm-up and the
  measurement sized independently within one process — which -warmup alone could not
  express.

• lith-s3bench -warmup, and the measured window's offset from process start
  (#381). A discarded window of the given
  length before the measured one, in the same process, and a start+NNNms field in the output
  so a cold first arm is self-identifying in a log.

  The first s3bench process of a run has been measured with a 3–10× worse first-byte tail in
  7 of 7 runs, with the idle gap beforehand ranging from seconds to ~8 h — which rules out
  a simply-cold endpoint. The decisive datum: a first s3bench process read endpoint 97.7 ms
  while a lith mount on the same box, same key, same instant read 41.2 ms. That rules out
  the endpoint, the network path, and box-wide state like DNS or the NIC, and leaves something
  in or keyed to the process's first contact.

  So this is both the mitigation and a diagno...