Pre-processing, segmentation, and comparative or individual analysis of MERSCOPE and Xenium spatial transcriptomics datasets.
The MerXen pipeline was designed to enable standardised analysis of MERSCOPE and Xenium datasets collected from brain tissue sections. MerXen allows for either standalone analysis of data from either one of these two spatial transcriptomic platforms, or for direct comparative analysis.
In the comparative analysis mode, pairs of adjacent brain tissue sections, one run on Vizgen MERSCOPE and the other on 10x Genomics Xenium, using the same gene panel are analysed together. MerXen re-derives cell boundaries from the raw imagery and transcripts on both platforms, brings the two sections into a common coordinate system, and runs an identical downstream analysis on each — so the differences measured are platform differences, not data processing difference from the pipelines provided by the two vendors.
By default every samplesheet row is a paired experiment, but the same workflow runs single-platform with --analysis_mode merscope or --analysis_mode xenium. In either mode, multiple samples can be run at once with the same parameters, each undergoing segmentation, per-cell image quantification, QC, cell-type assignment and an array of optional downstream analysis stages.
- Segmentation from scratch, optimised for brain tissue. Cellpose-SAM nuclei and cell segmentation, refined by ProSeg 3.2.0 transcript based segmentation using Cellpose as a prior — so both platforms derive cell boundaries in the same way.
- Matched downstream stages. Every analysis after ingest is platform-agnostic and runs identically on MERSCOPE and Xenium, including against the original instrument segmentations for direct comparison.
- Cross-section registration. Optional VALIS DAPI-morphology alignment puts adjacent sections in one coordinate system, with its own post-alignment QC.
- Reproducible by construction. Nextflow processes, isolated conda environments per conflicting stage, pinned lockfiles to reproduce the same analysis on any system with sufficient computing resources.
| Requirement | Notes |
|---|---|
| Linux | The pipeline is developed and tested on Linux. It may not run on macOS or Windows as-is and hasn't been tested on these platforms. |
Nextflow >=23.04 |
On your PATH. |
| Conda or Miniforge | Creates the merxen environment and the isolated per-stage environments. |
| Rust/Cargo or a ProSeg 3.2.0 binary | MerXen searches /usr/bin, /usr/local/bin, then command -v proseg, and builds the pinned revision with Cargo if none is found. |
| NVIDIA GPU | Strongly recommended. Cellpose-SAM falls back to CPU (--cellpose_gpu false) but is very slow on full sections. |
| Substantial RAM | Individual segmentation processes request up to 220 GB. See Configuration to dial requests down. |
git clone https://github.com/bourdenxlab/MerXen.git
cd MerXen
# 1. Environment (installs Python 3.12 and the merxen CLI)
conda env create -f envs/environment.yml
conda activate merxen
# 2. Environment variables
cp .env.example .env
# 3. Samplesheet — edit with your own dataset paths
cp workflows/samplesheet.example.csv workflows/samplesheet.csv
# 4. Run
nextflow run workflows/main.nf \
--samplesheet workflows/samplesheet.csv \
--outdir ./resultsFull walkthrough: Getting started.
Each stage is one Nextflow module and one merxen subcommand, sharing a Pydantic config contract.
| Stage | What it does | Default |
|---|---|---|
| SpatialData build | Builds platform-specific SpatialData zarrs from raw MERSCOPE and Xenium output folders | always |
| Segmentation | DAPI-only Cellpose nuclei → GPU Cellpose-SAM cells → ProSeg 3.2.0 refinement from Cellpose logits | always |
| Segmentation — hybrid | Transcript-supported local-convex branch with growth-only boundary smoothing | on (proseg_hybrid_enabled) |
| Enrichment | Shape layers and per-shape gene tables | always |
| Mask image quantification | Quantifies every SpatialData image channel over the final Cellpose masks | always |
| Cortical depth | Laplace / equal-area cortical-depth coordinates from boundary annotations | off (--cortical_depth_enabled) |
| QC | Per-dataset geometry and transcript-assignment metrics | always |
| MECR | Reference-based mutually exclusive co-expression rate against the WHB reference | on (--mecr_enabled) |
| Alignment | VALIS DAPI registration of paired adjacent sections, plus alignment QC | off (--enable_alignment) |
| Comparison | Cross-platform gene-level comparison | paired rows only |
| Visualization | Single-platform or paired figure generation | always |
| Spatial gene analysis | Cell-level spatial autocorrelation and assignment-independent transcript-coordinate patterns | always |
| Squidpy clustering | First-pass Scanpy/Squidpy clustering (RAPIDS-backed) | always |
| MENDER | Independent CPU-only MENDER spatial-domain analysis | off (--mender_enabled) |
| MapMyCells | Local Allen Institute MapMyCells cell type assignment | off (past default stop_stage) |
| Distance from object | Nearest registered polygon-edge annotation and grey-matter paired near-vs-far PyDESeq2 | off (--distance_from_object_enabled) |
Downstream analysis runs for both ProSeg-resegmented cells and the original instrument segmentations by default (--analysis_segmentation both); all adds the proseg_mask and proseg_hybrid branches.
See Metro map for what the diagram above does and does not show, and Pipeline architecture for the precise stage graph.
| Flag | Default | Description |
|---|---|---|
--samplesheet |
required | Path to your CSV. |
--outdir |
./results |
Where all outputs are published. |
--analysis_mode |
paired |
paired, merscope, or xenium. Controls which columns are required and which stages are active. |
--analysis_segmentation |
both |
both, all, reseg, original_seg, proseg_mask, or proseg_hybrid. |
--enable_alignment |
false |
Run VALIS alignment and alignment QC before comparison. Paired mode only. |
--alignment_backend |
valis |
valis, or legacy_spateo for the former expression-based implementation. |
--start_stage / --stop_stage |
build_spatialdata / clustering_squidpy |
Run a contiguous stage range. |
--only_stage |
— | Alias for setting start_stage and stop_stage to the same value. |
--cellpose_gpu |
profile-dependent | Set false to force CPU segmentation. |
Most of these can also be set per samplesheet row — a non-empty cell overrides the command-line value for that row only. Full parameter reference: Configuration.
Important
Nextflow selects the reserved standard profile when no -profile flag is given, and MerXen maps standard to the dwight workstation config. Passing any explicit profile suppresses standard, so -profile conda alone drops Dwight's executor capacity, concurrency guards, GPU locking, and reference paths. Combine them: -profile dwight,conda.
| Profile | Purpose |
|---|---|
standard / dwight |
Dwight workstation: 72 CPUs / 640 GB local executor, concurrency guards, shared GPU lock, local reference paths. |
conda |
Resolves each process against the repository's conda environments. |
apptainer |
Runs every process in the prebuilt CUDA 12.6 container. |
gpu |
Adds --nv to the GPU-bound processes only, keeping CPU-only containers GPU-free. |
azure_slurm_hpc |
SLURM executor, htc queue, 24 h wall time. |
local |
Dwight settings under the local executor. |
Other hosts must supply their own executor capacity, concurrency limits, GPU handling, and reference paths. More in Running the pipeline.
Each row points at raw platform folders, with optional reusable SpatialData cache paths and per-platform channel, z-range, and voxel-layer settings. Row-level columns can override most run defaults for a single sample, and object-distance runs supply registered object GeoJSON paths per platform. In single-platform rows, only the selected platform's columns are required.
A template lives at workflows/samplesheet.example.csv. The full schema, validation rules, and worked examples are in Samplesheet format.
results/
├── nextflow/ # report.html, timeline.html, trace.tsv
├── mecr_reference/ # Shared MECR marker discovery
├── <pair_id>/
│ ├── merscope/ # spatialdata, segmentation, enrichment, …
│ ├── xenium/ # …same stages, run independently
│ ├── alignment/ # + alignment_qc/
│ ├── reseg/ # mecr, comparison, visualization, clustering, …
│ └── original_seg/ # …the same analyses on instrument segmentations
└── distance_from_object/ # Cohort-level paired DE
Every .png is also written as a same-stem .pdf. Nextflow's own ./work/ directory is cache state, not output — deletable between full runs, but required for -resume. Every directory and file is documented in Outputs.
Full documentation lives in docs/ — start at docs/index.md.
- Usage — Getting started · Samplesheet format · Running the pipeline · Configuration · Outputs
- Developer reference — Pipeline architecture · Metro map · Python API · CLI reference · Development workflow
Repository layout
MerXen/
├── workflows/ # Nextflow pipeline
│ ├── main.nf # DSL2 entry point
│ ├── nextflow.config # Parameters, executor, per-process resources
│ ├── conf/ # Per-host profile configs
│ └── modules/ # One .nf module per pipeline stage
├── src/merxen/ # Installable Python package
│ ├── config.py # Pydantic configs (pipeline contract)
│ ├── cli/ # Click entry points (one per stage)
│ ├── io/ # Samplesheet, SpatialData builders, image/transcript I/O
│ ├── segmentation/ # Cellpose tiling + ProSeg subprocess
│ ├── enrichment/ # Shape layers + per-shape gene tables
│ ├── qc/ # Per-dataset and cross-platform metrics
│ ├── analysis/ # Scanpy/Squidpy downstream analyses
│ ├── visualization/ # Plotting
│ ├── cortical_depth/ # Laplace/equal-area depth coordinates
│ ├── distance_from_object/ # Polygon-edge distance + paired pseudobulk DE
│ └── alignment/ # Default VALIS DAPI + legacy Spateo registration
├── tests/ # pytest suite, mirrors src/merxen/
├── docs/ # Project documentation (start at docs/index.md)
├── notebooks/ # Exploratory notebooks only
├── envs/ # Base env + isolated alignment / clustering-GPU / MENDER envs
├── containers/ # Base, clustering-GPU and MENDER image definitions
├── requirements*.lock # Pinned dependency trees
└── Agents.md # Project standards (must-read for contributors)
Stages with conflicting dependency stacks run in their own environments: envs/environment.alignment.yml pins the VALIS 1.2 image-registration stack for ALIGN, envs/environment.clustering-gpu.yml provides RAPIDS, and envs/environment.mender.yml isolates MENDER's AnnData 0.9 / Scanpy 1.9 requirement. Everything else uses envs/environment.yml.
Project standards — layout, dependencies, naming, type hints, docstrings, git workflow, commit prefixes — are defined in Agents.md and apply to human and AI contributors alike. Do not commit to main; use a feature branch and open a PR.
pre-commit install
pre-commit install --hook-type pre-push
ruff check . --fix && ruff format . # lint and format
mypy src/ # type check
pytest # fast tests (excludes slow)
pytest --run-slow # include slow integration tests
scripts/run_ci_checks.sh # the same lockfile-backed checks as CIDay-to-day mechanics — testing, hooks, CI, debugging, adding a new pipeline stage — are in Development workflow.
MerXen orchestrates the work of a lot of other people. If you use this pipeline, please cite their works:
Core pipeline — Nextflow · Cellpose · ProSeg · SpatialData · Scanpy · Squidpy
Optional stages — VALIS · PyDESeq2 · MapMyCells · MENDER
Diagrams — nf-metro
MIT — see LICENSE. © 2025 Mathieu Bourdenx.
Developed in collaboration with the Bourdenx Lab.