Viewer transfer efficiency: v3 opens, probe sizing, chunk dedup, cache write-through

[neuromechanist]

Data-transfer and smoothness audit of the signal viewer's zarr read path, with the website-side wins implemented. Producer/edge-side findings are filed separately in nemar-cli (link below once filed) — the biggest lever (view-level chunk granularity) lives there, not here.

All numbers are real measurements against live zarr.nemar.org, driving the actual store.ts reader through an instrumented fetch (bun; no browser HTTP cache, which matches the worst case — level-0 Range responses are never browser-cached anyway).

Audit numbers (before)

Store A: on005514 sub-NDARAC589YMB_task-DespicableMe_eeg.zarr — 129 ch, 250 Hz, 172 s, 4 view levels.
Store B: on005514 sub-NDARXZ850KHQ_task-contrastChangeDetection_run-1_eeg.zarr — 129 ch, 250 Hz, 2430 s (40.5 min), 6 view levels.

Operation	Requests	Bytes	Notes
openRecording, store A	5 (2× 404)	57 KB	.zattrs + .zgroup 404 serially before zarr.json (zarrita's generic open tries v2 first on a fresh store); 3.6 s wall from here
openRecording + view discovery, store B	36 (≈20× 404)	67 KB	6-level pyramid probed in fixed batches of 6: batch 2 (levels 7–12) = 18 wasted requests, each miss tripled by the v2 fallback; 9.9 s wall
10 s window @ 1000 px, level 0	4	439 KB	shard index (suffix Range) + 3 inner chunks ≈146 KB each; all 206 responses — never cached by browser or edge
same window again	3	438 KB	full re-transfer (206s bypass every cache)
10 s window at view L1 / L2 / L3 / L4	3 each	230 / 63 / 19 / 5.5 KB	full GETs — edge- and browser-cacheable (max-age=86400)
full-recording window, store A (L2)	44	889 KB	~20 KB/chunk
full-recording window, store B (L4)	594	1.16 MB	~2 KB/chunk, 6.4 s wall
overview minimap, store B (L6)	148	77 KB	~0.5 KB/chunk — view chunks shrink to [2, 129, 1] at coarse levels

The request storms in the last three rows are producer chunk-shape issues (constant seconds per chunk instead of constant columns) — filed upstream, not fixable in the reader.

What this PR changes (with after numbers)

1. v3-first opens with a v2 escape hatch (openNode in store.ts). The fast path is open.v3: the store contract is Zarr v3 only, and zarrita's version-sniffing open cost 2 serial 404 round-trips before first metadata on every viewer open (~1.9 s wall measured) and tripled every probe miss. A NotFound falls back to open.v2 only until the store's first successful v3 read (a store is entirely one format), so the legacy safety net never re-adds the 404 pairs on a working v3 store. Caveat on scope: the v3-only observation was measured on two stores from one dataset (on005514), both from the same converter generation — hence the escape hatch rather than a bare pin.
2. Size the first view-level probe batch from n_samples (new pure predictedViewLevelCount; openGroup passes its already-resolved sample count, so the 12-probe fallback is unreachable in practice). The constants encode producer conventions verified on the two measured stores; prediction only sizes the batch — discovery still verifies every level, so a producer change degrades to a couple of extra probes, never a wrong level list. A non-404 probe failure (retry-exhausted 5xx, expired-token 403) now throws ViewLevelDiscoveryError carrying the levels that did open; the group keeps them and sets viewLevelsDegraded, so "discovery broke" is distinguishable from a genuinely short pyramid.
3. In-flight request dedup (dedupingFetch, wrapped outside retryingFetch in makeStore): one shared fetch per URL+Range across the interactive path and the Viewer: optional full-recording background streaming with capped cache #254 preloader. Level-0 chunks are 206 responses the browser cache never stores, so an overlap was a full ~140 KB re-transfer per chunk. Shared request aborts only when every subscriber has aborted; each subscriber gets an independently consumable clone.
4. Write interactive windows through to the preload cache via the pure writeThroughKey gate (prefetch.ts, sharing prefetchCacheKey with the walk's keyFor): a grid-aligned interactive read is the segment the background walk would fetch — without this, enabling preload re-fetched the exact window the user was already looking at on every (re)target (~430 KB per 10 s level-0 page, starting with the first window of every open). readFrame captures every geometry input before the awaited read so mid-flight scrolling can never store one window's data under another window's key, and a refused put (window larger than the whole cap) warns instead of silently disabling the optimization.
5. Save-Data guard: loadPreloadEnabled() returns false when navigator.connection.saveData is set — a browser-level "reduce data" preference outranks a stored opt-in from a previous session. Chromium-only signal; absence — or a hardened browser making the accessor itself throw — degrades to "no signal".

Operation	Before	After
openRecording, store A	5 req (2× 404), 3.6 s	3 req, 0 404s, 2.3 s
open + discovery, store B	36 req (≈20× 404), 9.9 s	15 req (2× 404), 6.6 s
view probing, store A (4 levels)	10 req (6 wasted)	6 req (2 wasted)
preload of an already-viewed 10 s page	~430 KB re-fetched	0 (cache hit)
interactive + preloader racing one chunk	2× ~146 KB	1× ~146 KB
window/pyramid byte costs	unchanged	unchanged (already viewport-bounded)

A side effect of (1): the hover/click metadata warmup (prefetchZarrStoreMetadata) now actually shortens the critical path — before, the open's first two requests (.zattrs/.zgroup) were unwarmed, uncacheable 404s sitting in front of the warmed URLs.

Evaluated and deliberately not changed

• pickViewLevel tuning: measurements show it already picks the coarsest level satisfying ≥1 sample/px, and level-0 selection is bounded to <16 s windows at 250 Hz (~439 KB) by the existing LEVEL0_MAX_SAMPLES cap. Reducing level-0 over-fetch further (2.5–4 samples/px worst case) needs denser pyramid spacing — a producer decision, noted upstream.
• Preloader schedule: it already targets the level the interactive path last rendered, yields to interactive reads and hidden tabs, walks outward from the playhead, and is byte-capped and opt-in (off by default). Preloading minutes the user never approaches is the documented, capped Viewer: optional full-recording background streaming with capped cache #254 design; no change justified beyond the Save-Data guard.
• Events read: already deferred off the first-paint path; its arrays are three tiny cacheable GETs.

Test plan

• bun run test 1379/1379, typecheck 0 errors, lint clean, build green.
• 32 new it() blocks, all boundary fakes per repo testing policy (no business-logic mocks):
  • predictedViewLevelCount — 5 (pinned to the two measured stores' real pyramids plus edge cases);
  • dedupingFetch — 10 (clone independence, partial/total abort lifecycle, post-abort and post-rejection rejoin, pre-aborted caller, and dedupingFetch(retryingFetch(...)) composition sharing one retry cycle between concurrent callers);
  • view-level discovery — 3, driven through the real reader pipeline against an in-memory fake Zarr v3 store (exact prediction stops after one follow-up confirm batch; undershot prediction creeps to the real end; a 403 probe keeps fulfilled siblings and sets viewLevelsDegraded); also asserts zero v2 fallback requests on a v3 store;
  • writeThroughKey — 6 (key equality with prefetchCacheKey for the same segment, off-grid/partial-trailing/disabled nulls, oversized put leaves resident entries intact);
  • preload settings — 8 (saveDataRequested / loadPreloadEnabled precedence both ways, throwing navigator.connection accessor degrades instead of crashing).
• Reader re-driven against live zarr.nemar.org after the changes (numbers above re-verified on the review-fix tip: open 3 req / no 404s, dedup confirmed at 4 requests for two concurrent identical level-0 window reads).

[cloudflare-workers-and-pages]

Deploying nemar-website with    Cloudflare Pages

Latest commit:	3526a44
Status:	✅  Deploy successful!
Preview URL:	https://286d1744.nemar-website.pages.dev
Branch Preview URL:	https://perf-viewer-transfer-audit.nemar-website.pages.dev

View logs

[neuromechanist]

Producer/edge-side companion filed: nemarOrg/nemar-cli#1178 (view-level chunk shape, declared pyramid, Range edge-caching, tokened-metadata TTL, index.json weight).

[neuromechanist]

Review-panel dispositions — all 16 findings addressed in e0e371a, 2c6ed05, 3526a44 (gates green on the tip: 1379/1379 tests, typecheck 0 errors, lint clean, build green; live re-verification against zarr.nemar.org shows identical request counts to the pre-review after-numbers).

CRITICAL

1. Write-through cache poisoning — fixed (2c6ed05). The gate + key derivation moved into a pure exported writeThroughKey(params) in prefetch.ts (item 9's helper); readFrame captures chanStart/chanCount/windowLengthS/preloadEnabled into locals before the awaited read and passes only those. prefetchCacheKey moved to prefetch.ts beside it so the walk's keyFor and the write-through provably share one key scheme.

HIGH

2. Throwing navigator.connection accessor — fixed (3526a44). saveDataRequested wraps the access in try/catch returning false, with a comment naming the fingerprinting-countermeasure scenario; a throw now degrades to "no signal" instead of taking down the mount. Covered by a test.
3. Non-404 probe failure folded into "pyramid ends here" — fixed (e0e371a). Fulfilled sibling probes are collected before any failure is examined (nothing discarded), and a non-404 failure throws a new ViewLevelDiscoveryError carrying partialLevels with the original failure on cause. openGroup keeps the partial levels and sets a new viewLevelsDegraded: boolean on GroupHandle, so callers can distinguish "discovery broke" from "genuinely short pyramid". Surface chosen: typed error + handle flag (documented on both); a clean 404 end stays a normal return. Tested via the fake-store harness (403 on view/2 → levels [1,3] kept, flag set).
4. v2 fallback — both done. (a) New openNode helper at every pinned call site: v3 first, NotFound-only fallback to open.v2, with a per-store latch (WeakSet) that disables the fallback after the store's first successful v3 read — a store is entirely one format, so this keeps the escape hatch without re-adding the two 404s per genuine miss that pinning removed (the fake-store test asserts zero .zattrs/.zgroup/.zarray requests on a v3 store). (b) PR body softened: the v3-only observation is now stated as measured on two stores from one dataset, same converter generation — hence the escape hatch.

MEDIUM

5. Discarded put() boolean — fixed (2c6ed05). A refusal (single window larger than the whole cache cap) now logs a console.warn naming the window size and the cap.
6. Abort/status mislabeling — clarified (e0e371a). The pre-review message did already carry both facts (returned ${status} (request aborted)); it now states them unambiguously — "aborted before retry (the status may be a real backend error, not an effect of the cancellation)" — and the comment names the dedup refcount-abort race explicitly.

TESTS

7. Sequential-after-rejection — added. Same key, first inner fetch rejects, next call gets a fresh inner fetch (calls === 2).
8. Save-Data precedence both ways — added (viewer.test.ts). saveDataRequested/loadPreloadEnabled exported; boundary-fake navigator/localStorage (descriptors saved/restored): Save-Data on + stored "1" → false; Save-Data off + stored "1" → true; connection-less browser + stored "1" → true; throwing accessor → no crash, falls through to stored value; nothing stored / no storage → false. 8 tests.
9. Pure write-through helper + tests — done. writeThroughKey takes every input explicitly (the docstring states the capture-before-await contract and why); tests cover key equality with prefetchCacheKey for the same segment, off-grid null, clamped-trailing-window null, disabled null, and an oversized put leaving resident entries and byte accounting intact.
10. Probe-loop harness — added. An in-memory fake Zarr v3 store (real-shape zarr.json documents; the full makeStore dedup+retry pipeline and openNode run unmodified against it): exact prediction probes predicted levels plus exactly one follow-up confirm batch and nothing more; prediction undershot by 2 creeps to the real end in VIEW_PROBE_FOLLOWUP steps.
11. Composition order — added. dedupingFetch(retryingFetch(6,1)), 503-then-200, two concurrent callers → exactly 2 inner fetches (one shared retry cycle), both callers get the 200.
12. Test-count claim — reconciled. The body's test plan now enumerates the actual additions: 32 new it() blocks across store/prefetch/viewer test files, and the suite total updated to 1379.

COMMENTS

13. VIEW_PROBE_FOLLOWUP doc — rewritten. Both the constant's doc and the loop comment now say the follow-up runs whenever the previous batch came back fully present — an exact prediction included — and that its job is confirming where the pyramid really ends.
14. Docstring placement — fixed. The discovery docstring sits on discoverViewLevels; isNotFound has its own line (tagged NotFoundError preferred, message regex as the non-zarrita fallback).
15. n_samples derivation — code fix taken. discoverViewLevels now takes nSamples as a parameter and openGroup passes its already-resolved value (attrs with the level-0-shape fallback), so the prediction is always a real number and the 12-probe worst case is unreachable in practice; docstring adjusted.
16. useSuffixRequest coupling — documented. The dedup docstring now states that "the reader issues nothing but GETs" is guaranteed by makeStore passing useSuffixRequest: true (without it, suffix reads become a HEAD + ranged GET and the HEAD would bypass dedup un-shared), and says to keep the two options together.

Verified-sound items (refcounted abort lifecycle, Range keying, dedup map self-heal, misprediction degradation, Save-Data precedence design, byte accounting, VERSION_COUNTER analysis): no changes, per the panel.