In order to better understand and evaluate segmentation methods, several methods and their variants have been chosen and compared. On the same 2mm × 2mm Xenium breast tissue, 33 method variants report anywhere from 11% to 40% of cells as luminal epithelial and differ five-fold in how often transcripts from one cell type bleed into neighboring cells. This benchmark tests all 33 methods against three independent reference standards (the platform's own segmentation, H&E tissue morphology, and matched single-cell RNA-seq) and finds that each standard rewards a substantially different set of methods.
Question: Which segmentation methods best preserve biological signal in spatial transcriptomics data?
Nine representative methods (the same ones in the Method Scorecard below), ranked 1 (best) to 29 among all tested methods against each of the three reference anchors described below. The lines cross constantly: Voronoi (M) is 4th-best against the platform's own segmentation (B1) but drops to 19th against H&E morphology (B2). Cellpose cyto3 density is 20th against the platform but 1st against H&E. No method stays near the top across all three - which anchor you trust changes which method wins.
For complete analysis and extended results tables, see FULL_PROJECT.md.
The evaluation has two parts. The first is a factorial design: each method is split into a nuclear detection step and a cell expansion step so the two can be measured independently. Four nuclear detectors (CellPose, StarDist, Mesmer, 10x Ranger) are paired with five expansion strategies (geometric 10/20µm, Voronoi, watershed, Baysor PSC sweep). Whole-cell NN methods (Cellpose cyto3, Mesmer WC) are tested separately.
The second is a three-anchor benchmark:
| Anchor | Information source | Strength | Blind spot |
|---|---|---|---|
| B1: 10x native | Proprietary DAPI-based algorithm | Practical standard; what most users see | Black-box expansion; biases invisible to B1 |
| B2: H&E morphology | DAPI + eosin whole-cell segmentation (Mesmer WC) | Real cytoplasm boundaries from tissue | Pre-IF registration artifacts; z-overlap |
| B3: scRNA-seq reference | Dissociated single-cell data, GSE243275 | True single-cell expression profiles | Dissociation bias; no spatial context |
When the three anchors agree, confidence is high. When they disagree, the disagreement pattern shows what each anchor is actually measuring. A method that scores well on B1 but poorly on B2 is reproducing the platform's DAPI-expansion logic, not tissue morphology.
Xenium FFPE Human Breast (Custom Add-on Panel), Janesick et al. 2023 (dataset page). Invasive ductal carcinoma; 2mm × 2mm ROI (~23,600 cells, ~3.4M transcripts, 380-gene panel). Registered H&E image provides hematoxylin (DAPI surrogate, r = 0.896) and eosin (cytoplasm signal) channels via skimage.color.rgb2hed(). Matched scRNA-seq + Visium from the same tissue blocks: GEO GSE243275.
| Family | n | Variants |
|---|---|---|
| 10x native | 1 | Platform reference segmentation (also evaluated as a test method) |
| Nuclear-only | 4 | CellPose, StarDist, Mesmer, 10x Ranger |
| Voronoi | 4 | CP / SD / M / 10x Ranger seeds |
| Geometric 10µm | 4 | CP / SD / M / 10x Ranger |
| Geometric 20µm | 4 | CP / SD / M / 10x Ranger |
| Watershed | 3 | 10x / SD / M seeds, DAPI gradient |
| Baysor, no prior | 1 | No nuclear seeds (PSC = 0) |
| Baysor, CP seeds (PSC sweep) | 4 | CP prior at PSC 0.2 / 0.5 / 0.8 / 1.0 |
| Baysor, PSC 1.0 (other seeds) | 3 | SD / M / 10x Ranger seeds at PSC 1.0 only |
| Cellpose cyto3 | 3 | DAPI / DAPI+eosin / DAPI+density |
| Mesmer WC | 2 | DAPI+eosin / DAPI+density |
Nuclear-only methods capture 35–52% of transcripts and are excluded from downstream figures past the recovery section.
In progress: Proseg (dcjones/proseg), a Rust transcript-density segmenter with an explicit leaky-RNA diffusion model, is installed and wired into the method registry but not yet run - not counted in the 33 above, and absent from every results table/figure until its baseline run completes. See FULL_PROJECT.md for details.
Mesmer produces the best nuclear detections regardless of expansion strategy (Voronoi M: ARI 0.686 vs. 10x native; Baysor M prior: ARI 0.518). 10x Ranger (purpose-built for Xenium) matches CellPose and StarDist on nuclear quality alone; its 99% transcript capture in the 10x native output comes from its proprietary expansion, not from larger nuclei.
B1 ARI for every detector × expansion-strategy combination with complete data (Watershed × CellPose was never run - nuclear-only methods are excluded throughout, see "Methods Tested" below). Mesmer's row stays dark across every column: detection quality survives the choice of expansion strategy. The same expansion strategy varies far more across detector rows than any one detector varies across columns - this is the factorial design's point, made visible.
For Baysor, PSC acts as a threshold rather than a gradient. Below PSC 0.2, cell counts, ARI, and violation rates are nearly identical to the no-prior baseline. At PSC 0.8 and above, cell counts jump to ~30,000, matched pairs nearly double, and Moran's I rises from 0.036 to 0.219, converting spatially diffuse disagreement into structured disagreement at tissue boundaries.
Luminal epithelial cells (pink) under three methods. The ductal clusters visible in the left two panels dissolve under Baysor without a prior (right) - the PSC=0 condition described above.
Voronoi methods reproduce 10x native cell-type composition within 1–3 percentage points and reach ARI 0.58–0.69 against B1. Against the H&E anchor (B2), the ranking reverses: Cellpose cyto3 DAPI+density reaches ARI 0.807 and Mesmer WC density 0.776, while 10x native scores only 0.590, lower than 26 of 33 tested methods. Methods with a positive B2 vs. B1 delta (whole-cell NN, Baysor prior, StarDist-seeded expansion) track morphology; methods with a negative delta (Voronoi CP, Voronoi M, geometric with 10x Ranger) track the platform standard.
Spearman rank correlations across 28 non-anchor methods: B1 vs. B3 ρ = 0.832 (p < 0.0001), B1 vs. B2 ρ = 0.139 (p = 0.49), B2 vs. B3 ρ = 0.006 (p = 0.98). B1 and B3 are largely redundant; both reward high transcript capture. B2 is orthogonal to both.
Every method plotted by B2 (x) and B3 (y, inverted so up-and-right = better on both). Whole-cell NN methods (green) lead on B2 but sit apart on B3; the Baysor PSC sweep (pink) trails badly on B3 regardless of B2; nuclear-seeded expansion methods (blue) cluster tightly near the 10x native anchor (gray) on B3 across a wide spread of B2 values. B2 and B3 are measuring genuinely different things, matching the near-zero Spearman correlation above.
In the scRNA-seq reference space, nuclear-seeded expansion methods and 10x native cluster together (centroid distance 3.47–3.58). Whole-cell NN methods form an intermediate tier (3.68–4.30), and Baysor PSC ≥ 0.8 sits at 4.22–4.38 while Baysor PSC ≤ 0.5 lands at 4.61–4.65, a gap that holds regardless of detector.
Eleven Tier 1 marker pairs (e.g., CD3E + GATA3, MS4A1 + KRT14) define biologically impossible co-expression events; a cell expressing both members of any pair almost certainly contains transcripts from two adjacent cells merged by the segmentation. Cellpose cyto3 DAPI achieves the lowest violation rate at 0.77% (0.14 per 1000 transcripts), three to eight times lower than any Voronoi or geometric method. Voronoi methods and 10x native cluster together at 4.49–6.45%, while Baysor PSC ≥ 0.8 falls at 2.17–2.34%. High B1 ARI is weakly negatively correlated with low violation rate: methods that best reproduce 10x native clustering tend to have higher cross-lineage contamination, because both are reproducing the same boundary placement errors.
Left: per-method violation rates across all 29 non-nuclear-only methods. Right: violation rate vs. median transcripts per cell, labeled at the 6 highest, 6 lowest, and 10x native; whole-cell NN methods (lower right) achieve lower violations at comparable or higher transcript capture.
Same 29 methods, same top-to-bottom order as the bar chart above, broken down by which marker pair drove each violation. LYZ+GATA3 (macrophage/luminal) and CD3E+GATA3 (T cell/luminal) drive most of the signal, together accounting for roughly three-quarters of all Tier 1 violations averaged across methods; the other 9 pairs are comparatively rare.
The bleed analysis scores T cells by mean expression of CAF marker genes and maps which T cells sit near CAF-rich stroma. Under every centroid-based and gradient method (including 10x native), roughly 38–52% of T cells score above the high-bleed threshold, with median T-cell-to-CAF distances of 15–33µm. Mesmer WC density is qualitatively different: only 9.6% of its T cells are high-bleed, with a median T-cell-to-CAF distance of 131µm. High-bleed T cells under Mesmer WC are not adjacent to CAF boundaries; the violations that remain are not a proximity artifact. This four-to-five-fold reduction over every other method, including Cellpose cyto3, is the largest spatial signal in the benchmark. The cluster 3 contamination test supports this: methods with broader expansion (Voronoi CP: CAF-to-luminal marker ratio 3.69) pull proportionally more stromal transcripts into luminal cells than Baysor prior methods (M prior: 2.90).
T cells (colored circles) scored by mean CAF marker expression, overlaid on CAF centroids (gray). Under centroid-based methods, high-bleed T cells saturate the stromal boundary across the full tissue. Under Mesmer WC (DAPI+density), 9.6% of T cells are high-bleed vs. 38–52% under all other methods, and the remaining violations are not spatially clustered at boundaries.
Nine methods (B1/B2 ARI vs. 10x native/H&E; B3 cell-type centroid distance to scRNA-seq, lower = better; negative-marker violation rate, lower = better), values printed on each cell. Each column is shaded independently by its own value, darker = better within that column - colors aren't comparable across columns, and because shading follows the actual value rather than rank, the top four B3 scores (3.48–3.51, effectively tied per the main text) still span most of that column's color range; read the printed numbers for those, not just the shade. 10x native is darkest on B1 by definition (it's the anchor) but lightest on B2; Cellpose cyto3 DAPI+density is darkest on B2 but among the lightest on B1 and B3. Full 31-method table in FULL_PROJECT.md.
One repo, one Python package (segbench), one Snakefile, three pipeline
stages sharing one conda env (segan - originally three separate repos,
each with its own env, recombined here - see docs/dataset.md/TEST.md for
the history). PyTorch/Cellpose (segimg) and scanpy/squidpy/scikit-learn
(segan) install fine into one env. StarDist and Mesmer share a second env
(mesmer): Mesmer's deepcell hard-requires tensorflow~=2.8.0, and it
turns out that same TensorFlow satisfies StarDist too (StarDist doesn't pin
a TensorFlow version at all) - verified by re-running StarDist through the
merged env and diffing the output mask pixel-for-pixel against the one from
StarDist's former standalone env: identical foreground, identical per-cell
areas (see TEST.md):
- segimg (
src/segbench/segimg/) - raw Xenium bundle + H&E in, per-method label masks out. Also reaches into Docker (Mesmer nuclear) and Julia (Baysor), neither of which is a conda env at all. - segcore (
src/segbench/segcore/) - masks + raw transcripts in, one frozenSpatialData.zarrobject per method out. - segan (
src/segbench/segan/) -SpatialDatain, comparison tables + figures out.
Nothing in src/ imports across stages (segimg/segcore/segan never
import from each other) - the three stages only communicate through files
under data/. That property is what originally made running them in three
separate conda envs possible, and it's also why merging those three envs
into one was safe: no cross-stage import surface to break.
segmentation-benchmark/
├── environment.yml, environment.lock.yml # segan env (all three stages: PyTorch/Cellpose, scanpy/squidpy/spatialdata, scikit-learn, ...)
├── mesmer-environment.yml # mesmer env: Mesmer (deepcell) + StarDist, sharing deepcell's TensorFlow 2.8 pin
├── Snakefile # all three stages' rules, in one file (segimg -> segcore -> segan order)
├── configs/ # Baysor *.toml configs
├── docs/dataset.md # dataset description, ROI selection, format gotchas
├── reference_data/ # scRNA-seq companion (GSE243275), tracked in git - small and needed for the B3 anchor
├── data/ # raw bundle, masks, SpatialData .zarr - gitignored, not tracked in git
├── src/segbench/
│ ├── segimg/ # image -> mask: io.py, segmentation/ (Cellpose/StarDist/Mesmer/Baysor wrappers)
│ ├── segcore/ # mask -> SpatialData: quantify.py, build.py
│ └── segan/
│ ├── constants.py # method metadata, cell-type annotations, negative marker pairs
│ ├── load.py # load_method()/method_exists() - reads data/spatialdata/<method>.zarr
│ ├── compare.py # cross-method comparison metrics
│ ├── spatial.py # spatial structure of disagreement
│ ├── style.py # shared matplotlib theme
│ └── figure_qa.py # pixel-heuristic figure QA (see TEST.md)
├── scripts/ # CLI entry points, one per method family/analysis
├── results/{figures,tables}/
└── tests/ # flat, one shared conda env - test_code_hygiene.py, test_io.py, test_build.py, test_compare.py, ...
# segan (shared env - all three stages: CellPose/geometry/I/O, mask+transcript
# quantification, comparison tables/figures)
conda env create -f environment.yml && conda activate segan
# mesmer (shared env - Mesmer whole-cell/in-process nuclear, and StarDist)
conda env create -f mesmer-environment.yml
docker pull vanvalenlab/deepcell-applications:latest # Mesmer nuclear via Docker
juliaup add 1.10 && julia +1.10 -e 'using Pkg; Pkg.add(PackageSpec(url="https://github.com/kharchenkolab/Baysor.git", rev="v0.7.1")); Pkg.build("Baysor")' # Baysorsnakemake --cores 4 # everything, all three stages end to end
snakemake --cores 4 all_segimg # just the segimg stage
snakemake --cores 4 all_segcore # just the segcore stage
snakemake --cores 4 all_segan # just the segan stage
snakemake -n # dry runTwo segan-stage rules, local_morans and de_disagree, write a dynamic
per-method/per-matcher set of files with nothing fixed to declare in
rule all_segan, so a full snakemake --cores 4 won't trigger them
automatically - run them by explicit rule name instead:
snakemake local_morans de_disagree. Every other stage (resolution
sensitivity, the rest of spatial disagreement, sensitivity/drivers, pairwise
consensus/marker recovery) is already wired into rule all.