Add spatial_expression dataset_type with sidecar target-matrix plumbing#263
Merged
Conversation
Introduce a multi-target regression supervision shape so downstream HEST slices can curate and probe gene-expression targets without touching the schema again. - SUPERVISION_COLUMN gains spatial_expression -> target_index - write_manifest accepts target_matrix/genes, validates them, and emits targets.npy + genes.json sidecars beside dataset.csv (uncompressed .npy / plain .json keep re-writes byte-identical) - SampleRecord carries a resolved vector `target` (compare=False) - SpatialExpressionManifest loader reads the sidecars, fail-fast validates them, and attaches each spot's vector; load_manifest dispatches to it
6 tasks
clemsgrs
added a commit
that referenced
this pull request
Jul 9, 2026
Turn a pre-provisioned hest-bench task tree (per-slide patch HDF5 + gene-expression .h5ad + per-fold slide split CSVs + a task gene list) into soma's spatial_expression Manifest, building on #263's shared write_manifest / target-matrix sidecar plumbing. - Explode each slide's img[N] patch array into per-spot lossless PNGs referenced by image_path (preserving HEST's exact pixels). - Bake targets = log1p(raw counts) on float64 for the task genes (no total-count normalization, no smoothing) into the targets.npy + genes.json sidecars. - Expand each HEST fold's slide-level train/test membership to per-spot rows (spot_id, "train"|"test", fold=i); no tune rows. - Emit a curator-authored summary.json; re-curation is byte-identical. - Curation deps (anndata, h5py) go in a new soma[hest] optional extra, lazy-imported inside curate_hest so import soma stays light and core CI does not require them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A sanctioned prefactor that teaches soma's Manifest a multi-target regression supervision shape, so later HEST slices can curate and probe gene-expression targets without touching the schema again ("make the change easy, then make the easy change"). No curator and no probe here — only the schema shape and its load path.
Introduces a new
dataset_type—spatial_expression— whose single supervision column istarget_index, an integer row-key into two sidecars written besidedataset.csv:targets.npy— shape[n_rows, n_genes], rowiis thetarget_index==ispot's vector.genes.json— the ordered gene list.One sample = one spot. The loaded
SampleRecordcarries its resolved vector ontarget;SpatialExpressionManifestattaches it from the sidecar.Acceptance criteria
spatial_expressionis a recognizeddataset_typewithtarget_indexas its supervision column in the sharedSUPERVISION_COLUMNmap.write_manifestemits a validspatial_expressionManifest (dataset.csvwithsample_id,image_path,target_index) plus thetargets.npyandgenes.jsonsidecars; the loader rejects a Manifest missing the sidecars or the column (fail-fast).spatial_expressionManifest yields sample records whose vector targets round-trip exactly (values and gene order) from the sidecar.dataset_types (tile/slide/patient/segmentation/detection) are unaffected.Serialization choices
targets.npy(uncompressed.npyvianp.save,allow_pickle=False, C-contiguous). Uncompressed keeps re-writes bit-deterministic;np.savez_compressed(used elsewhere for lossy heatmaps) is not byte-stable across zlib versions. Dtype is preserved verbatim from the caller (the design bakes float64 log1p counts).genes.json(json.dumps(list(genes), indent=2)— order preserved, never sorted, since gene order is load-bearing).CuratedManifest(target_matrix_path/genes_path,Nonefor other dataset_types). Filename constants live insoma.datasetand are imported by the writer to keep writer/reader in lockstep.SampleRecord.targetiscompare=False(derived supervision data, not sample identity; array fields have no scalar truth value).Validation added
Writer and loader both fail-fast on: a missing sidecar, a non-2D matrix, a gene/column-count mismatch, and out-of-range
target_index. Passingtarget_matrix/genesfor any non-spatial_expressiondataset_type is rejected.Deviations / notes
PipelineConfig._valid_dataset_types(insoma/config.py) is intentionally not extended in this slice. There is no probe/task yet to make aspatial_expressionpipeline runnable, and adding it would force premature aggregator/task constraints. The load path (load_manifest) — this issue's scope — recognizes it; the runnable-pipeline wiring lands with the probe (design §10, PR 2).hestextra or HEST curation dependency added (deferred, per the issue).Tests
Added 11 tests in
tests/test_curation_manifest.pycovering the supervision-column map, sidecar emission + round-trip, byte-identical re-write, loader dispatch/round-trip, missing-sidecar / missing-column rejection, and the writer's range/count/misuse guards.Full suite:
python -m pytest -q→ 1513 passed, 3 skipped, 1 xfailed.Closes #257