seqFISH functionality; v0.6.0; beta - #48
Merged
Merged
Conversation
Xenium writes its multi-channel stack to morphology_focus/, but list_images only scanned the dataset root, so those channels were unreachable from the image picker even though the files were sitting right there. Both sample datasets ship four of them. list_images now scans the root and one level of subdirectories, returning bare filename stems. _find_source resolves a stem back to a path by searching the same two locations in the same order, so the picker can never list an image the tile builder cannot open. Root-level files are added first and stems are de-duplicated, so a root file wins any name collision with a subdirectory file in both functions. Hidden directories are skipped so .dzi_cache is never scanned. Salvaged from a GitHub Desktop stash (28616a9) whose edge-file changes were superseded by the merged RaredonLab#46 implementation; this part had never landed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d plans; v0.4.0 CLAUDE.md had drifted far enough to mislead. Corrections: - Visium HD was entirely absent despite being a registered reader (second in detection order). Added it, plus a note that VisiumHDReader.transcripts and .cell_boundaries still carry the pre-refactor limit= signature and would raise TypeError if the capability flags did not stop the frontend calling them. - Per-panel rotation (RaredonLab#31) and RenderingStatus.jsx were undocumented. Added a Rotation section covering the two places the angle must be applied, why the fetch bbox is padded, and why viewportActual exists separately. - The cloud deployment story was missing entirely: docs/cloud-deploy.md, Caddyfile, deploy.sh, docker-compose.prod.yml, upload-data.sh. Corrected the claim that there is no auth at all — Caddy basicauth exists but is opt-in and off by default. - Fixed the r/ listing, which named a file deleted in b30bc82, and the App.jsx path. Documented the actual three-script pipeline. - Replaced the stale fracW zoom-skip thresholds with the sampling fractions that actually govern fetch volume now. - Reordered What's Not Built Yet to lead with the real bottleneck: spatial reads pull whole parquet files into pandas on every viewport change instead of pushing the bbox down to DuckDB the way the edge path does. Added the CellInfoPanel supplemental-metadata gap and open issues RaredonLab#35 and RaredonLab#45. README: replaced your-lab placeholder URLs with RaredonLab, added Visium HD to the platform table with an honest note on per-platform completeness, documented multi-channel morphology, multiple edge sets, split-screen and rotation, linked the cloud-deploy runbook, corrected export_for_TissuePlex to export_to_TissuePlex, and added a roadmap pointing at the open issues. vite.config.js defaulted the dev server to port 3000 — the same port docker compose binds — which is why launch.json had to override it. Set it to 5173 so the config matches the documented intent and npm run dev stops colliding with a running container. PLAN.md, PLAN_v2_2026-05-02.md and EDGE_UI_PLAN.md move to OBS/ with a README explaining why each is obsolete. EDGE_UI_PLAN in particular specifies the opposite of what shipped on both edge aggregation and lrm_set coloring. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both methods loaded their entire parquet with pq.read_table(...).to_pandas()
and then masked in pandas, so every viewport change materialized the whole
file to keep a viewport-sized slice of it.
Measured on a synthetic 40M-row / 0.78 GB transcripts.parquet, one zoomed-in
viewport query:
pandas full read + mask 2903 MB peak RSS 912 ms
DuckDB streaming 233 MB peak RSS 1369 ms
12x less memory. That is the binding constraint: production runs on a 16 GB
droplet, where a multi-GB transcripts.parquet under the old path exhausts RAM
long before it is merely slow.
DuckDB is somewhat slower here, and the reason is worth recording. Xenium
writes transcripts.parquet in row order, not spatial order, with very large
row groups -- the bundled breast dataset is 1.1M rows in 2 row groups, the
first spanning the entire x-range. Row-group statistics are therefore useless
and DuckDB scans everything anyway, paying predicate-evaluation cost with none
of the pruning payoff. Sorting the file spatially and rewriting it with small
row groups takes the same query from 1010 ms to 29 ms; that is a follow-up,
noted in CLAUDE.md.
Also fixes polygon clipping. Boundaries previously filtered individual
vertices by bbox, so cells straddling the viewport edge came back missing
part of their outline and rendered as torn shapes -- 97 such cells on the
bundled breast dataset. Selection is now per cell: a cell qualifies if any
vertex is in the bbox, and all of its vertices are returned. Sampling likewise
draws whole cells.
Shared query helpers live in readers/duck.py so the other platform readers can
adopt the same path. Verified against pandas ground truth on both bundled
datasets: identical totals for full-extent, bbox-quadrant and gene-filtered
queries, identical pixel-space conversion, and sampling that is deterministic
across repeated calls.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design proposal only -- no reader implemented yet. Records the format research and the decisions taken: target Spatial Genomics GenePS rather than academic seqFISH+ (which has no standard output), one ROI per folder, bare integer cell labels as cell_id, and an edge-metadata/ folder mirroring the existing cell-metadata/ convention. The format claims are verified against the real scverse CI fixture rather than taken from documentation. Two findings changed the plan: A single seqFISH dataset mixes coordinate systems. Measured on the 1000x1000 DAPI at 0.107161 um/px: CellCoordinates center_x spans 1.82-105.66 and TranscriptList x spans 0-107.05, both microns, while Boundaries.geojson vertices span 0-999, pixels. Cells and transcripts need dividing by pixel_size and boundaries must pass through untouched. One global transform in either direction would put cells and their own outlines in different places, which presents as a rendering bug rather than a unit bug. The px-vs-um heuristic (ratio of max coordinate to image width) separates all three cases here with two orders of magnitude of headroom. GeoJSON features carry the cell label in their `id` field -- strings "1".."62" matching CellCoordinates.label exactly. spatialdata-io does not read this and maps polygons to cells positionally, with an open issue about the fragility (scverse/spatialdata-io#249). Joining on id is correct by construction; a silent off-by-one there would draw every outline on the wrong cell. sample_data/.gitignore excludes seqfish*/ (and visium-*/). The test dataset is public for CI by written permission from Spatial Genomics, not under an open licence, so it must not be redistributed from this repo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Groundwork for seqFISH support. Two changes, neither altering Xenium behaviour. backend/tests/golden_snapshot.py exercises every reader method against the bundled datasets, digests the results, and diffs them against a recorded baseline -- 62 probes across the two Xenium datasets, covering info, capabilities, gene lists, cells, schema, transcripts (full/bbox/gene-filtered/ sampled), boundaries (full/sampled), per-cell detail and expression, colour values, and every edge endpoint for each edge file. It calls readers directly rather than over HTTP, so no server is needed and failures point at the reader instead of the transport. This repo has no test suite, so cross-platform reader work had nothing to catch a regression. Verified the guard actually bites: flipping duck.SAMPLE_SEED from 42 to 43 fails exactly the six sampled probes and passes again on revert. Two sources of false failure had to be removed first. Record-list digests are now order-independent, because query_grouped uses ORDER BY RANDOM() and SQL GROUP BY promises no ordering -- hashing in returned order failed on every run. And the edge_detail probe picked grouped[0], a different edge each time; it now takes the lexicographic minimum. _load_supplemental_metadata and _read_csv_with_barcodes move from XeniumReader to SpatialDatasetReader, so cell-metadata/ becomes available to every platform rather than Xenium alone. Nothing platform-specific was in them; all Xenium contributes now is the list of its own root CSVs to ignore. That list is the trap this opens, so it is closed here too: MERSCOPE and CosMx both write CSVs at the dataset root that the shared loader would otherwise ingest as user metadata. MERSCOPE gets an exact-name skip set; CosMx prefixes every output with the experiment name, so it needs suffix matching, which _is_platform_csv now supports. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a fifth platform reader. Xenium is untouched: all 62 of its golden-snapshot
probes are byte-identical across the change.
"seqFISH" names two unrelated things. The academic Cai-lab method has no
standard output layout; this targets the commercial Spatial Genomics GenePS
platform, which does. Current v2 layout is fully supported -- cells,
transcripts, boundaries, expression, both colour-value modes. Legacy v1 reads
cells and transcripts but declares has_boundaries: False, since v1 ships only a
label mask and polygonising it would mean either a heavy new dependency or
hand-rolled contour tracing; deferred deliberately.
The hard part is coordinates. A single seqFISH dataset mixes units, measured on
the reference dataset (1000x1000 DAPI at 0.107161 um/px):
CellCoordinates center_x 1.82 -> 105.66 microns
TranscriptList x 0.00 -> 107.05 microns
Boundaries vertices 0 -> 999 pixels
Cells and transcripts are divided by pixel_size; boundaries pass through
untouched. One global transform in either direction would put cells and their
own outlines in different places, presenting as a rendering bug rather than a
unit bug. The convention also differs across GenePS software versions, so it
cannot be hard-coded: _units_divisor decides per table by comparing that
table's extent to the image width, and logs its verdict. On real data the
ratios are 0.106 / 0.107 / 0.999 -- two orders of magnitude apart.
Verified geometrically rather than by digest alone: every cell centroid falls
inside its own polygon, 62/62 on the reference dataset and 36/36 on the
synthetic fixture, with zero false positives against a control. Independently,
polygon area recomputed from boundary vertices agrees with the reported
cell_area to within 3%.
Cell identity comes from each GeoJSON feature's id, which equals label.
spatialdata-io maps polygons positionally instead and has an open issue about
the fragility (scverse/spatialdata-io#249); a silent off-by-one there would
draw every outline on the wrong cell.
sample_data/make_seqfish.py generates a committable synthetic v2 ROI (400K) and
deliberately reproduces the mixed units, so a reader that got them wrong would
fail on it. The real reference dataset is public by written permission from
Spatial Genomics rather than under an open licence, so it stays gitignored.
Three bugs fixed along the way, all pre-existing and none seqFISH-specific:
- CellInfoPanel rendered the literal string "undefined um2" for any platform
not reporting a cell area -- the optional chain was not paired with a null
check the way nucleus_area's is.
- Switching datasets kept the previous dataset's activeImage until the image
list resolved, so OSD spent that window requesting the old image from the new
dataset and logging 404s. Only visible now because image names differ across
platforms ("morphology" vs "Roi1_DAPI"). setDataset clears it and OSD init
waits.
- cell_area is kept in um2 to match Xenium, which never converts it, so the
"um2" label in the info panel is true on every platform.
Verified end-to-end in the browser across Xenium -> seqFISH -> Xenium with zero
console errors and no backend errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The edge-side mirror of cell-metadata/: annotate cell pairs -- a call, a
confidence, a review flag -- without regenerating edges.parquet from R.
dataset_dir/
edge-metadata/
annotations.csv key column `edge` = "SendingCell|ReceivingCell"
Rather than write a second loader, the cell-metadata implementation moves to
readers/supplemental.py parameterised on the key column, and both features now
share it. Same file types, same outer-join, same forgiving key resolution
(explicit key column -> "Unnamed: 0", pandas' name for R's unnamed rowname
column -> first column if unique strings), so the R default just works. Moving
it is behaviour-preserving: all Xenium probes stayed identical across the
refactor.
The folder sits beside the dataset, not beside the edge file. _dataset_dir
walks up out of edges/ when the edge file is nested there, so one set of
annotations applies across every edge source in the dataset -- annotations
describe cell pairs, which are a property of the tissue rather than of one
scoring run. Verified against both edges.parquet and edges/*.parquet.
Three integration points in edge_reader.py, and no frontend change was needed
for the main one: schema() merges supplemental columns into the returned map,
and LayerPanel builds the edge colour-by dropdown straight from the schema, so
they appear on their own. edge_color_values() checks the parquet first then the
supplemental frame, skipping the GROUP BY since supplemental data is already
one row per edge. edge_detail() attaches matches under a `metadata` key.
Parquet wins a name collision -- a supplemental column silently shadowing a
real one would be painful to debug.
EdgeInfoPanel renders those generically as an "Annotations" block above the LRM
table, so any column the user adds shows up untouched.
mouse_ileum_tiny gains worked examples of both cell-metadata/ and
edge-metadata/. Neither feature had a sample dataset before, so cell-metadata
had been unexercisable locally since it shipped.
Verified in the browser: the three annotation columns appear in the edge
colour-by dropdown, a categorical one renders its legend and colours the edges,
and clicking an edge shows the annotations alongside its LRM scores --
interaction_class "heterotypic" on a Fibroblast -> Endothelial edge, confirming
the join lands on the right row. No console or backend errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Version bump covering seqFISH platform support, the reader regression guard, and supplemental edge metadata. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.