Hive virtual store: moczarr-backed /gridlook/hive/ (phase 6d of issue #1)

[espg]

Refs #1 (phase 6d)

The reserved /gridlook/hive/ namespace becomes real: a moczarr-backed virtual store that serves an open_hive() product/AOI/window selection as ONE flat zarr v3 store the browser's unmodified zarrita reads. This is the cryocloud-MVP slice from the MVP map: post-englacial/zagg#314 stores are morton-only, so moczarr's fabricated NESTED cell_ids (fabricate_cell_ids="auto") is what makes them renderable at all — it rides every view served here.

What this adds

• GET /gridlook/hive/open?store=<url>[&product=…][&aoi=<decimal,csv>][&window=…] — creates (or LRU-refreshes) a session-scoped view, returns {view, url, cells, cell_order, cached}; the url is the zarr source to paste into gridlook.
• GET /gridlook/hive/<view-id>/<zarr-key> — serves the view's zarr objects (metadata documents + whole small chunks; no Range needed).
• Auth posture inherited from phase 4: @web.authenticated everywhere, s3:// stores gated by the same GridlookProxy.allowed_buckets allowlist (403 otherwise), local paths only behind the new allow_local_hive_stores dev flag.
• New traits: hive_max_views (LRU bound, default 8), hive_max_cells (per-view bound, default 500 000 — a pointed 413 tells you to narrow the AOI), allow_local_hive_stores.
• gridlook-jupyter[hive] extra carrying moczarr @ git+https://github.com/espg/moczarr (until PyPI; tool.hatch.metadata.allow-direct-references gates the direct reference). The base install stays moczarr-free; hive.py imports it lazily and the open endpoint 500s with a pointed install hint when it's absent.

Materialize-on-open (the implementation lean, justified)

open_hive(...) → ds.to_zarr(zarr.storage.MemoryStore(), zarr_format=3, consolidated=False) per view, LRU-bounded. Chosen because: (a) views are AOI-scale and bounded by hive_max_cells, so the memory cost is capped and known (hive_max_views × hive_max_cells); (b) xarray writes the store, so the serve path contains zero zarr chunk/codec arithmetic — the handler serves opaque objects via the store's async get; (c) every request after open is a dict lookup. Chunks are written uncompressed (compressors: None): objects go whole over hub-local HTTP, and codec-free bytes are trivially predictable (the tests compare them raw). A streaming/virtual encoding — computing zarr objects on demand from the open dataset — is documented in hive.py as the future optimization if views outgrow memory; the URL contract wouldn't change.

View ids are deterministic (hash of the request tuple), so re-opening the same selection refreshes the existing view instead of duplicating it; serving objects LRU-bumps the view, so actively-rendered views survive eviction pressure.

Pre-6c compatibility shim (explicit, temporary)

The served root attrs do not advertise the stored morton convention block ({name: "morton", coordinate: "morton"}, mortie spec §5) — gridlook's gridTypeDetector.ts rejects any dggs.name != "healpix" today. Instead the view serves the healpix-compatible block the detector and Healpix.vue read now: name: "healpix", refinement_level (→ nside), coordinate: "cell_ids". That is honest at the data level — the fabricated NESTED ids are plain HEALPix NESTED indices at that refinement level, bit-for-bit what the existing sparse limited-area path consumes — so the MVP renders without phase 6c. The shim is marked in hive.py (_shim_dggs_attrs) and goes away when 6c lands the morton detector branch.

Tests (50 pass locally; hive suite skips without moczarr)

pytest-jupyter, mirroring the phase-4 suite, against moczarr's committed SERC fixture (zagg-written, morton-only) as a local-path store:

• open → entry URL + cells; zarr.json carries the shimmed healpix attrs block
• fabricated cell_ids chunk byte-equals serc_cell_ids_golden.npy (the last dual-written store's stored NESTED ids — fabricate-at-serve loses nothing)
• AOI subsetting (one shard → 16 cells; ids match fabricate_cell_ids of the served morton words)
• 403 non-allowlisted bucket, 403 local-path when disabled, 404 unknown/evicted view, 404 missing object, 400 bad aoi/window, 413 over hive_max_cells, LRU eviction incl. refresh-bumps-recency
• fixture discovery: moczarr repo checkout next to the installed package (or GRIDLOOK_MOCZARR_TESTDATA); plain [test] envs skip the module wholesale — existing CI is untouched and stays green

Gates run: pytest -q → 50 passed (env: uv venv && uv pip install -e '.[test]' -e /Users/espg/software/moczarr); plain [test]-only env → 35 passed, 1 module skipped; ruff check + ruff format --check clean.

MVP recipe (documented in jupyter/README.md)

1. pip install <wheel> "gridlook-jupyter[hive]" on the hub image/user env
2. c.GridlookProxy.allowed_buckets = ["my-zagg-outputs"] (or GRIDLOOK_ALLOWED_BUCKETS), restart the server
3. GET …/gridlook/hive/open?store=s3://my-zagg-outputs/store-root&aoi=4331422
4. open …/gridlook/, paste the returned view url as the dataset source — renders through the existing sparse-HEALPix path

Questions for review

• CI: .github/workflows/jupyter.yml installs only [test], so the hive suite skips there (left untouched per the no-CI-edits convention). If you want it exercised in CI, the job needs a moczarr checkout + editable install step — say the word and it lands as a follow-up.
• hive_max_cells default (500 000) is sized as ~40–50 MB/view at SERC-like variable counts; happy to retune.
• The [hive] extra's git pin is unversioned (@ git+… HEAD); once moczarr tags/publishes, it becomes a normal moczarr>=X pin and allow-direct-references drops.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N32Mx4rWhbTK8xXm7Mt2ax

[espg]

🤖 from Claude (review)

Unconstrained local-path oracle behind the dev flag. With allow_local_hive_stores=True, store= reaches any local path — probed /etc, /etc/passwd, ~/.jupyter all get past the gate and into open_hive(). Two problems on a multi-user hub:

1. The 400/404 bodies echo moczarr's messages verbatim ({e} at L#L245/L250-ish), which distinguish '/etc/passwd' is not a directory (file exists) vs no morton_hive.json at /etc (dir exists, no manifest) vs a missing path → a filesystem existence/type oracle for an authenticated user.
2. Any real hive store anywhere on the box (another user's $HOME, a shared scratch dir) is openable and servable — cross-user read, not just a dev convenience.

Default-off + documented-dev-only mitigates, but it's a full local-file reach the moment the flag flips. Suggest a rooted-directory trait (an allowlist of permitted roots + os.path.realpath containment check) so "dev" can't silently become whole-filesystem access. Probed with the test client; not blocking, but worth a rooted variant.

[espg]

🤖 from Claude
Replaced the boolean with a rooted-directory trait local_hive_store_roots (empty = disabled): a local path is now accepted only when its os.path.realpath is contained in a configured root, and anything outside gets a single uniform 403 ("not within an allowed root") with no existence/type-distinguishing body, closing the filesystem oracle — d6dd5f2.

[espg]

🤖 from Claude (review)

refinement_level is taken from cell_order with no float64-exact / point-kind guard. The frontend computes nside = 2**refinement_level (Healpix.vue L185) and does all cell math in JS doubles. But moczarr's fabricate_cell_ids clips POINT-kind words to order 24 (FLOAT64_EXACT_MAX_ORDER) and only warns for AREA cells above 24 — and that warning is emitted on the executor thread, so it's swallowed, never reaching the operator or the 200 response. Consequences the fixture (order 8) never exercises:

• cell_order > 24 area store: served cell_ids are exact u64 the browser reads as lossy Numbers, and 12*nside*nside overflows 2^53 → silently wrong render, HTTP 200, no signal.
• point-kind store where cell_order ≠ 24: shim advertises refinement_level = cell_order (e.g. 29) while the fabricated ids are clipped to order 24 → declared nside ≠ served ids' order → cells placed at the wrong scale.

Since this store exists only to feed a float64 browser consumer, recommend guarding cell_order > 24 at open (pointed 400/413) rather than serving an unrenderable/mismatched view. At minimum, document the ceiling here.

[espg]

🤖 from Claude
The shim now derives refinement_level from the SERVED ids (point words clip to order 24, so a point store declares 24 and renders correctly instead of at a mismatched nside=2**29), and area stores whose ids exceed the float64-exact range (order > 24, or any id ≥ 12·4²⁴) are rejected at /open with a pointed 422 rather than served — 0ad2923.

[espg]

🤖 from Claude (review)

View id isn't normalized → trivial LRU blowup / redundant materialization. Probed against the SERC fixture:

• aoi=4331421,4331422 vs aoi=4331422,4331421 → different view ids, identical 32 cells
• aoi=4331421,4331421,4331422 (dup) → a third id, same 32 cells
• store=…/serc_hive vs …/serc_hive/ (trailing slash) → different ids, same store

Every distinct id is a full re-open (S3 GETs + concat) and consumes an LRU slot, so a client varying token order / duplicates / trailing slash multiplies work and evicts legitimately-open views — an easy way to thrash the cache out from under other users. build_view already rstrips the root for the data path, but the hash keys on the raw store_url + raw aoi tuple. Canonicalize before hashing: rstrip store_url, and sorted(set(int(t) for t in aoi)) so order/dupes collapse to one id.

[espg]

🤖 from Claude
view_id now canonicalizes before hashing — rstrip the store, normalize empty params to absent, and sorted(set(int(t) ...)) the aoi — so reordered/duplicated tokens and a trailing slash all collapse to one id (unit-tested) — 89d9fa7.

[espg]

🤖 from Claude (review)

The advertised memory bound covers resident views only; concurrent in-flight builds are uncapped. Real footprint on the fixture is 52 B/cell → 26 MB per 500k-cell view, ~208 MB across 8 (so the config's ~100 B/cell is a safe 2× over-estimate — good). But build_view is dispatched to the default ThreadPoolExecutor (run_in_executor(None, …), up to ~min(32, cpu+4) threads) with no concurrency limit, and each in-flight build holds the xarray ds plus the MemoryStore copy transiently. N concurrent large-view opens ⇒ N×(ds+store) on top of the 208 MB resident cache — so "hive_max_views × hive_max_cells bounds the cache footprint" (config help + PR body) understates peak on a 2–4 GB pod under a burst of opens. Consider a small bounded executor or a semaphore around opens, or at least document that the bound is resident-only.

[espg]

🤖 from Claude
Added a hive_max_concurrent_builds trait (default 2) and a per-loop module asyncio.Semaphore around the executor offload, so a burst of opens queues rather than holding N×(ds+store) transiently on top of the resident cache; the config help now says the cell bound is resident-only, and a test opens 4 at once and all succeed — 27862be.

[espg]

🤖 from Claude (review)

Shim rewrites the dggs block to healpix but leaves the stored morton-dggs entry in zarr_conventions. Probed the served root zarr.json: zarr_conventions carries both the generic dggs entry and the stored morton-dggs ("Packed-u64 morton (HEALPix) DGGS convention") entry, while the dggs block now says name: healpix, coordinate: cell_ids. Today's detector only checks that the zarr_conventions key exists then reads the dggs block (gridTypeDetector.ts determineGridTypeFromZarrConvention), so this is harmless now — but the served store is internally contradictory: it claims conformance to the morton-dggs convention while serving a healpix block against a cell_ids coordinate. A stricter/future validator that keys on the convention entries would trip. Since the shim already owns the block rewrite, consider dropping (or replacing) the morton-dggs entry here too so the envelope and the block agree.

[espg]

🤖 from Claude
The shim now drops the stored morton-dggs entry from zarr_conventions (keeping the generic dggs registry entry) so the served envelope and the healpix-flavored block agree, and preserves the store’s own morton block verbatim under a private _gridlook_source_dggs attr for provenance — 1077815.

[espg]

🤖 from Claude (review)

Fresh-context adversarial review of phase 6d (the moczarr-backed /gridlook/hive/ virtual store). Set up the worktree env per the PR body (uv pip install -e '.[test]' -e /path/to/moczarr) — 50 pass; all probes below were run against the SERC fixture via the test client.

Inline findings (severity-ordered)

1. allow_local_hive_stores is an unconstrained local-path oracle (medium) — reaches /etc, /etc/passwd, ~/.jupyter; error text distinguishes file/dir/missing; any user's on-box hive store is readable. Default-off/dev-only mitigates. Suggest a rooted-directory trait.
2. Shim refinement_level = cell_order with no >24 / point-kind guard (medium) — nside = 2**refinement_level in JS doubles; fabricate_cell_ids clips point words to order 24 and only warns (swallowed on the executor) for area cells >24. cell_order>24 or point-kind ⇒ silently wrong/mismatched render at HTTP 200. Fixture is order 8, untested.
3. View id not normalized (low-med) — probed: reordered aoi, duplicate aoi, and trailing-slash store each mint a distinct view id for identical data → redundant materialization + LRU thrash that evicts other users' views.
4. Memory bound is resident-only; concurrent builds uncapped (low-med) — real 52 B/cell → 26 MB/view, 208 MB @8 (config's ~100 B/cell is a safe 2× over-estimate). But run_in_executor(None, …) has no concurrency cap and each build holds ds+store transiently, so peak can exceed the advertised bound under a burst.
5. Shim leaves stored morton-dggs in zarr_conventions while the dggs block is rewritten to healpix (low) — served envelope is internally contradictory; harmless to today's detector (verified it only checks the key then reads the block), trips a stricter validator.

Minor (not posted inline)

• _parse_aoi doesn't cap token count — 5000-token aoi= accepted (200); output is gated by hive_max_cells so impact is a mild input-side DoS (regex + moc_and over the list). A token-count cap would close it.
• The 413 body suggests narrowing aoi= but not window=.

Verified sound

• Auth: @web.authenticated on both handler families; no-token requests serve no data (probed — request never completes at the gate, never 200).
• S3 allowlist enforced on store= (probed s3://sneaky-bucket → 403); non-s3:// scheme → 403; casing can't bypass (only lowercase s3:// hits the allowlist branch).
• No path traversal: product regex forbids / and a leading dot, so it can't escape the root; the <key> capture is a dict lookup in the MemoryStore, so ../… just 404s (no filesystem touch).
• Event-loop nesting is fine: build runs on an executor thread (no running loop → moczarr/zarr use their own background loop via sync()); serve awaits MemoryStore.get on the tornado loop (no loop affinity). Probed 4 simultaneous opens → distinct views, correct cells, no deadlock/cross-talk.
• Size gate fires on ds.sizes before any data load → 413 before materialize.
• LRU has no data race: get/put/serve all run on the single tornado loop; only build_view is offloaded and cache.put happens back on-loop.
• Uncompressed byte-equality of served chunks holds (the raw-compare tests are honest).

[espg]

🤖 from Claude

Acknowledging the two minor (non-inline) notes from the review summary:

• _parse_aoi token-count cap — not added as a hard cap, but the view-id canonicalization from finding (3) now sorted(set(...))-dedupes aoi tokens, so repeated tokens collapse before hashing/materialization (a duplicate-driven blowup no longer mints distinct views). A true count cap on distinct tokens is still open; leaving it for @espg since output is already gated by hive_max_cells (mild input-side cost only).
• 413 body naming only aoi= — addressed while in finding (2): the oversize message now reads "narrow the aoi= or window= selection" (27862be region / the 413 branch), so both narrowing levers are named.

[espg]

If you want it exercised in CI, the job needs a moczarr checkout + editable install step — say the word and it lands as a follow-up.

this would be nice, and is easier now that moczarr's on pypi...