Skip to content

Add spatial_expression dataset_type with sidecar target-matrix plumbing#263

Merged
clemsgrs merged 1 commit into
mainfrom
feat/spatial-expression-dataset-type
Jul 9, 2026
Merged

Add spatial_expression dataset_type with sidecar target-matrix plumbing#263
clemsgrs merged 1 commit into
mainfrom
feat/spatial-expression-dataset-type

Conversation

@clemsgrs

@clemsgrs clemsgrs commented Jul 9, 2026

Copy link
Copy Markdown
Owner

Summary

A sanctioned prefactor that teaches soma's Manifest a multi-target regression supervision shape, so later HEST slices can curate and probe gene-expression targets without touching the schema again ("make the change easy, then make the easy change"). No curator and no probe here — only the schema shape and its load path.

Introduces a new dataset_typespatial_expression — whose single supervision column is target_index, an integer row-key into two sidecars written beside dataset.csv:

  • targets.npy — shape [n_rows, n_genes], row i is the target_index==i spot's vector.
  • genes.json — the ordered gene list.

One sample = one spot. The loaded SampleRecord carries its resolved vector on target; SpatialExpressionManifest attaches it from the sidecar.

Acceptance criteria

  • spatial_expression is a recognized dataset_type with target_index as its supervision column in the shared SUPERVISION_COLUMN map.
  • The shared write_manifest emits a valid spatial_expression Manifest (dataset.csv with sample_id, image_path, target_index) plus the targets.npy and genes.json sidecars; the loader rejects a Manifest missing the sidecars or the column (fail-fast).
  • Loading a hand-built spatial_expression Manifest yields sample records whose vector targets round-trip exactly (values and gene order) from the sidecar.
  • Re-writing the same Manifest is byte-identical (deterministic).
  • Existing dataset_types (tile/slide/patient/segmentation/detection) are unaffected.

Serialization choices

  • Target matrix → targets.npy (uncompressed .npy via np.save, allow_pickle=False, C-contiguous). Uncompressed keeps re-writes bit-deterministic; np.savez_compressed (used elsewhere for lossy heatmaps) is not byte-stable across zlib versions. Dtype is preserved verbatim from the caller (the design bakes float64 log1p counts).
  • Gene list → genes.json (json.dumps(list(genes), indent=2) — order preserved, never sorted, since gene order is load-bearing).
  • Both sidecar paths are surfaced on CuratedManifest (target_matrix_path / genes_path, None for other dataset_types). Filename constants live in soma.dataset and are imported by the writer to keep writer/reader in lockstep.
  • SampleRecord.target is compare=False (derived supervision data, not sample identity; array fields have no scalar truth value).

Validation added

Writer and loader both fail-fast on: a missing sidecar, a non-2D matrix, a gene/column-count mismatch, and out-of-range target_index. Passing target_matrix/genes for any non-spatial_expression dataset_type is rejected.

Deviations / notes

  • PipelineConfig._valid_dataset_types (in soma/config.py) is intentionally not extended in this slice. There is no probe/task yet to make a spatial_expression pipeline runnable, and adding it would force premature aggregator/task constraints. The load path (load_manifest) — this issue's scope — recognizes it; the runnable-pipeline wiring lands with the probe (design §10, PR 2).
  • No hest extra or HEST curation dependency added (deferred, per the issue).

Tests

Added 11 tests in tests/test_curation_manifest.py covering the supervision-column map, sidecar emission + round-trip, byte-identical re-write, loader dispatch/round-trip, missing-sidecar / missing-column rejection, and the writer's range/count/misuse guards.

Full suite: python -m pytest -q1513 passed, 3 skipped, 1 xfailed.

Closes #257

Introduce a multi-target regression supervision shape so downstream HEST
slices can curate and probe gene-expression targets without touching the
schema again.

- SUPERVISION_COLUMN gains spatial_expression -> target_index
- write_manifest accepts target_matrix/genes, validates them, and emits
  targets.npy + genes.json sidecars beside dataset.csv (uncompressed .npy /
  plain .json keep re-writes byte-identical)
- SampleRecord carries a resolved vector `target` (compare=False)
- SpatialExpressionManifest loader reads the sidecars, fail-fast validates
  them, and attaches each spot's vector; load_manifest dispatches to it
@clemsgrs
clemsgrs merged commit c14cf35 into main Jul 9, 2026
3 checks passed
@clemsgrs
clemsgrs deleted the feat/spatial-expression-dataset-type branch July 9, 2026 11:59
clemsgrs added a commit that referenced this pull request Jul 9, 2026
Turn a pre-provisioned hest-bench task tree (per-slide patch HDF5 +
gene-expression .h5ad + per-fold slide split CSVs + a task gene list)
into soma's spatial_expression Manifest, building on #263's shared
write_manifest / target-matrix sidecar plumbing.

- Explode each slide's img[N] patch array into per-spot lossless PNGs
  referenced by image_path (preserving HEST's exact pixels).
- Bake targets = log1p(raw counts) on float64 for the task genes (no
  total-count normalization, no smoothing) into the targets.npy +
  genes.json sidecars.
- Expand each HEST fold's slide-level train/test membership to per-spot
  rows (spot_id, "train"|"test", fold=i); no tune rows.
- Emit a curator-authored summary.json; re-curation is byte-identical.
- Curation deps (anndata, h5py) go in a new soma[hest] optional extra,
  lazy-imported inside curate_hest so import soma stays light and core
  CI does not require them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spatial_expression dataset_type + sidecar target-matrix plumbing

1 participant