v0.7.0 — Visium HD + seqFISH support, NICHESv2 pipeline, spatial index

[msraredon]

v0.7.0 — Visium HD + seqFISH support, NICHESv2 pipeline, spatial index

Adds two platforms, a NICHESv2 workflow that produces real edge data, and a large performance change to the spatial query path. Xenium behaviour is unchanged throughout, and that claim is now checkable rather than asserted — see Regression guard below.

13 commits · 41 source files · +5,284 / −364 (excluding sample data and the recorded test baseline).

---

Platforms

seqFISH — Spatial Genomics GenePS

Full support for the current v2 layout: cells, transcripts, boundaries, expression and both colour-value modes. Legacy v1 reads cells and transcripts, and declares has_boundaries: False because v1 ships only a label mask.

The hard part is coordinates. A single seqFISH dataset mixes units, measured on the reference dataset (1000×1000 DAPI at 0.107161 µm/px):

Source	Extent	Units
CellCoordinates.csv center_x	1.82 → 105.66	microns
TranscriptList.csv x	0.00 → 107.05	microns
Boundaries.geojson vertices	0 → 999	pixels

Applying one transform to everything puts cells and their own outlines in different places, which reads as a rendering bug rather than a unit bug. The convention also differs across GenePS software versions, so it cannot be hard-coded. _units_divisor() decides per table by comparing that table's extent to the image width, and logs its verdict.

Verified geometrically rather than by digest: every cell centroid falls inside its own polygon — 62/62 on the reference dataset, 36/36 on the synthetic fixture, zero false positives against a control.

One improvement over the reference implementation: cell identity comes from each GeoJSON feature's id. spatialdata-io maps polygons positionally and has an open issue about the fragility (scverse/spatialdata-io#249); a silent off-by-one there would draw every outline on the wrong cell.

Visium HD

Built against a real Space Ranger 4.0.1 outs/ tree, not a hand-assembled folder — which mattered, because three separate things prevented genuine output from loading:

• Detection globbed the wrong level. square_*um/ lives under binned_outputs/, so the top-level glob never fired on real data.
• The tile pipeline was TIFF-only. Visium HD ships tissue_hires_image.png. Added PNG to source resolution plus a Pillow fallback, since the tifffile path cannot open one.
• The reader's method signatures did not match the base class (limit= instead of fraction=, returning [] instead of the documented dict), so a direct call raised TypeError. It was unreachable only because the capability flags stopped the frontend asking.

Bins render as square polygons. A bin is literally a square of side spot_diameter_fullres, so cell_boundaries() emits four vertices per bin instead of declaring has_boundaries: False. This matters because nothing renders cells() centroids — boundaries are the only path to drawing a unit — so a points-only reader would show an empty canvas. Emitting squares makes fill, outline, colour-by, picking and region selection all work with no frontend change at all.

sample_data/visium_hd_tiny/ is 10x's Tiny Mouse Brain dataset (CC BY 4.0, so redistributable), trimmed from the 297 MB download to ~26 MB.

---

NICHESv2 → edges.parquet

One script per platform in r/, each run against the demo datasets to produce real edge files. No button in the UI — deliberately out of scope.

Script	Distinctive part
niches_xenium.R	The simple case: coordinates already µm. Read first.
niches_seqfish.R	Dense cells × genes CSV needing transposition; units vary by GenePS version
niches_visium_hd.R	Coordinates are pixels, converted with microns_per_pixel
niches_common.R	Shared helpers only — 10x h5 reader, LR-coverage check, validation

The Visium HD conversion is the point of that script. export_to_TissuePlex() copies meta.data$x/$y verbatim into x1..y2 and the backend divides by pixel_size assuming microns, so raw pixels offset every edge from its bin silently. Verified by joining the exported edges back to tissue_positions: max |dx| = max |dy| = 0 px across all 3,321 bins.

Results on the demo data:

Dataset	Units	Edges	Scored rows	LRMs
xenium_human_breast_2fov	7,275 cells	181,523	189,063	32
visium_hd_tiny	3,321 bins	66,001	147	69
seqfish_synthetic	36 cells	250	7,364	30

Top Xenium hits are CDH1|CDH1, CXCL12|CXCR4, PTN|SDC4 — plausible for breast.

Two things running this for real forced:

• mouse_ileum_tiny cannot be used for NICHES, and that is the data not the script — its matrix is a near-empty placeholder (467 counts, 7 expressed genes), so no LR pair can score. It keeps synthetic edges from make_edges.py.
• The synthetic seqFISH panel was unusable. Gene00/Gene01/… match nothing in any LR database, so NICHESv2 aborted. make_seqfish.py now generates 36 real mouse symbols forming 20 verified pairs in connectomedb2025. check_lr_coverage() also now reports scorable pairs up front, so a genuinely unscorable panel explains itself instead of failing deep inside compute_CellToCell().

---

Performance

Transcript and boundary queries moved from "read the whole parquet into pandas, then mask" to DuckDB, then gained a spatial index. Measured on a synthetic 40M-row / 0.78 GB transcripts file, one zoomed-in viewport query:

	peak RSS	wall time
pandas full read + mask	2903 MB	912 ms
DuckDB streaming	233 MB	1369 ms
DuckDB + spatial index	233 MB	44 ms

12× less memory was the first win, and the binding one: production runs on a 16 GB droplet, where a multi-GB transcripts file under the old path exhausts RAM long before it is merely slow.

Speed needed two further steps. Xenium writes transcripts.parquet in acquisition order with very large row groups — the bundled breast dataset is 1.1M rows in two row groups, the first spanning the entire x-range — so statistics exclude nothing and DuckDB scans everything anyway. spatial_cache rewrites the file sorted by a coarse spatial grid with small row groups, built on first access and cached, the same pattern ensure_pyramid already uses.

That alone gave only 1.4×. DuckDB prunes row groups at plan time, and with bound ? parameters the filter values are unknown then, so it cannot prune:

sorted file, COUNT    literals 6.8 ms  vs  ? params 155 ms
unsorted file, COUNT  literals 186 ms  vs  ? params 178 ms

On an unsorted file the two are identical, which is why this never mattered before. Inlining the bbox as literals is what unlocked the 27×; it is safe because the values pass through float() with non-finite rejected, while string filters still bind.

Also fixed: boundaries previously filtered individual vertices by bbox, so cells straddling the viewport edge came back missing part of their outline — 97 cells on the bundled breast dataset rendered as torn polygons.

---

Cross-platform metadata

cell-metadata/ moved from XeniumReader onto the base class, so every platform gets it, and edge-metadata/ is new — annotate cell pairs without regenerating edges.parquet from R:

dataset_dir/
  edge-metadata/
    annotations.csv     key column `edge` = "SendingCell|ReceivingCell"

Both share one loader (readers/supplemental.py) parameterised on the key column, so the two conventions cannot drift. Columns appear in the edge colour-by dropdown automatically — no frontend change was needed, since the dropdown is built from the schema.

mouse_ileum_tiny now ships worked examples of both. Cell metadata had no sample dataset at all before, so despite being documented it could not be exercised locally.

---

Regression guard

backend/tests/golden_snapshot.py — 133 probes across 5 datasets, covering every reader method plus every edge endpoint for every edge file.

cd backend && python3 tests/golden_snapshot.py          # check
cd backend && python3 tests/golden_snapshot.py --record # adopt intentional changes

The repo had no test suite, so cross-platform reader work had nothing to catch a regression. Verified the guard actually bites: flipping duck.SAMPLE_SEED fails exactly the six sampled probes and passes again on revert.

Two sources of false failure had to be removed first — record-list digests are now order-independent (query_grouped uses ORDER BY RANDOM()), and the edge_detail probe picks the lexicographic minimum rather than grouped[0], which was a different edge each run.

---

Other fixes

• Multi-channel morphology — Xenium morphology_focus/ channels were unreachable; list_images scanned only the dataset root. Salvaged from an unmerged stash.
• CellInfoPanel rendered the literal string "undefined µm²" on any platform not reporting a cell area — the optional chain was not paired with a null check.
• Dataset switching logged 404s — activeImage kept the previous dataset's value until the image list resolved. Latent until now because every platform used morphology.
• vite.config.js defaulted the dev server to port 3000, the same port docker compose binds, which is why launch.json had to override it. Now 5173.
• export_for_TissuePlex() does not exist — corrected to export_to_TissuePlex() in four places, and docs/data_format.md checked against the actual writer: it always emits all 16 columns, and unscored edges get a placeholder row with null LRM/score fields (which is the origin of the null-LRM handling already in the backend).

---

Docs

CLAUDE.md and README.md brought back in line with the code — an entire platform (Visium HD), the cloud-deploy path, per-panel rotation and RenderingStatus were all undocumented, and the r/ listing named a file deleted several commits earlier. Superseded planning docs moved to OBS/ with a README explaining why each is obsolete; EDGE_UI_PLAN.md in particular specifies the opposite of what shipped.

Verification

Guard green at 133 probes. Verified end-to-end in the browser across Xenium → seqFISH → Visium HD → Xenium with no console or backend errors: bins render as a regular grid registered on the H&E, seqFISH outlines sit on their own cells, and NICHESv2 edges overlay correctly on both.

Notes for reviewers

• Sample data grew by ~26 MB (visium_hd_tiny, CC BY 4.0) plus ~4.5 MB of generated edge files. The seqFISH reference dataset is not committed — it is public for CI by written permission from Spatial Genomics rather than under an open licence, and sample_data/.gitignore excludes it.
• Still open: EdgeReader is not spatially indexed and still binds its bbox, so it would need the same literal treatment before an edge index would pay off. CellInfoPanel still renders a hardcoded field list rather than showing supplemental columns generically the way EdgeInfoPanel now does.