network_fmri

Orchestrator / runner for the r01network neuroimaging pipeline. It is the single source of truth for the BIDS dataset — it builds it (and, over time, downstream derivatives) by importing and running open-source packages. It is deliberately thin: the only study-specific code here is what no package can know — our Flywheel→BIDS heuristic + acquisition map + curation config.

What's in the box

The BIDS-curation engine is fw-heudiconv (our sherlock-compat fork, pinned to an immutable commit in pyproject.toml). This repo supplies only the study-specific pieces the engine needs:

file	role
src/network_fmri/heuristic.py	the fw-heudiconv heuristic (keys on the Flywheel acquisition label)
src/network_fmri/curation.py	byte-for-byte acquisition→BIDS map + config-derived aliases/overrides/skips
src/network_fmri/session_map.py	chronological ses-01.. renumbering + per-subject curate job plan
src/network_fmri/run.py	the fw2bids runner
config/curation_config.json	Flywheel block (aliases/overrides/skips) + cohort rosters

Companion packages (imported, not vendored)

• fw-heudiconv @sherlock-compat — the BIDS-curation engine (multi-echo/fmap file-selection, deterministic run numbering).
• network_qa — QA metrics/decisions (short-run flagging via nf-qa-runs); its verdicts feed the data-selection layer.

Documentation

• docs/SCAN-NOTES.md — curation-layer facts + source-level corrections applied on Flywheel, with a dated changelog (the authoritative record of what was acquired / relabeled / removed).
• docs/DATA-SELECTION.md — how exclusions partition across .bidsignore (invalid only) / bids-filter-file (processing selection) / scans.tsv (why), and why that replaces the legacy .bidsignore + symlink-farm.
• CONTRIBUTING.md — developer setup (container + uv venv overlay) for extending the package.

Quickstart — reproducible run (the container path)

The pipeline is designed to run through a container (network_fmri.sif): it bakes the pinned stack and a modern git-annex (≥ 10). This matters — the datalad and select stages use DataLad 1.6, which rejects Sherlock's host git-annex 8; the container is how those stages get a compatible git-annex. Build it once, then run the whole DAG with one command:

# 1. Build the image (compute node; ~35 min). Reads network_fmri.def in this repo.
sbatch -p normal -c 4 --mem 16G -t 01:00:00 --wrap="
  export APPTAINER_TMPDIR=\$SCRATCH/apptainer_tmp APPTAINER_CACHEDIR=\$SCRATCH/apptainer_cache
  cd \$HOME/network_fmri
  apptainer build --fakeroot --force \
    /home/groups/russpold/singularity_images/network_fmri.sif network_fmri.def"

# 2. Authenticate to Flywheel once (see step 3 below), then run the full DAG:
fw2bids pipeline --cohort discovery --container --staging $SCRATCH/bids_staging

--container (bare) uses the default image path above; --container <path> picks another. Everything below is the same DAG broken into per-stage detail; add --container to any fw2bids submit <stage> / fw2bids pipeline call to run that stage through the image. The uv-venv path in step 1 is for dry-runs, offline tests, and host-side rendering — but datalad/select must use --container.

End-to-end walkthrough (Sherlock, from scratch)

1. Build the runtime environment (on a compute node)

$HOME is NFS-quota'd, so the venv and uv cache live on $SCRATCH. numpy/pandas have no glibc-2.17 (CentOS 7) wheels, so uv sync compiles them from source — that must run on a compute node, never the login node:

sh_dev -c 8 -m 24000 -t 01:00:00 -p normal   # interactive compute shell (-m is MB)
module load uv
export UV_PROJECT_ENVIRONMENT=$SCRATCH/network_fmri_venv
export UV_CACHE_DIR=$SCRATCH/uv_cache
cd $HOME/network_fmri
uv sync                                       # ~7 min first time (compiles numpy/pandas)

Everything after this is light and can run on the login node (still with the two UV_* exports set, so uv finds the scratch env).

2. Run the offline unit tests

uv run --no-sync pytest -q                # 17 pure-stdlib tests, no Flywheel needed

3. Authenticate to Flywheel (once)

fw login <YOUR_API_KEY>                    # from flywheel.stanford.edu → profile

The key is stored under ~/.config/flywheel/, so it is shared across environments and only needs doing once.

4. Generate BIDS (fw2bids)

fw2bids is dry-run (read-only) by default — it computes the intended BIDS names without writing to the shared Flywheel project.

uv run fw2bids discovery                    # dry-run all discovery subjects
uv run fw2bids discovery --subject s03      # dry-run one subject
uv run fw2bids validation
uv run fw2bids excluded

4b. Write the BIDS directory (--live --out)

Materializing BIDS on disk is fw-heudiconv's two-step: curate persists the BIDS naming into each file's info.BIDS on the Flywheel project (a write — snapshot first), then export downloads the tagged files to a directory. fw2bids does both when given --live --out:

# snapshot the r01network project first, then:
uv run fw2bids discovery --live --out $SCRATCH/bids_staging/discovery   # tag + export

Stage on $SCRATCH and run bids-validator before writing the canonical Oak tree. (--live tags the shared project; --out requires --live.)

Curation is a one-time write per cohort; once done, export is a pure read and can be split subject-by-subject with fw2bids export:

uv run fw2bids export validation --subject s286 --out $SCRATCH/parts/s286

export never writes to Flywheel, retries transient download drops (--retries N), and takes one subject to its own dir — so a whole cohort export becomes an embarrassingly-parallel Slurm array (one small, independent, retryable job per subject) instead of a single fragile whole-roster download. Merge the per-subject parts into one tree afterward (rsync -a $SCRATCH/parts/*/ $DEST/).

4b-2. Merge per-subject exports (merge)

export writes one dir per subject under <staging>/parts/; the merge stage rsyncs them into the single cohort tree the downstream stages read (rsync -a <parts>/<cohort>/*/ <staging>/<cohort>/) — the stage form of the manual rsync noted above:

fw2bids submit merge --cohort discovery --container --staging $SCRATCH/bids_staging

4c. Trim dummy volumes (trim)

fw2bids export writes un-trimmed BIDS, but fMRIPrep is run with --dummy-scans 0, so the 7 non-steady-state volumes must be removed from the staged tree first:

uv run fw2bids trim $SCRATCH/bids_staging/discovery

Idempotent (a sidecar NumberOfVolumesDiscardedByUser flag skips already-trimmed files, so it's safe to re-run) and atomic (writes to a temp file, then renames over the original). --subjects s10 s19 ... restricts to those subjects, which lets a large cohort be sharded across a Slurm array with each task owning a disjoint set of files (no write races):

uv run fw2bids trim $SCRATCH/bids_staging/validation --subjects s10 s19

4c-2. Generate event files (events)

Turns the in-scanner behavioral logs into BIDS events.tsv sidecars, orchestrated by network_events. It migrates the reviewed behavioral CSVs from the read-only OAK raw dir into sourcedata/in_scanner_behavior/ (per the vendored reconciliation_<cohort>.tsv manifest), creates the per-run events.tsv (with dummy-volume onset adjustment baked in), runs behavioral QC, and writes truncation sidecars under sourcedata/events_qc/ (which the select stage's behavioral generator reads):

fw2bids submit events --cohort discovery --container --staging $SCRATCH/bids_staging

Skipped for the excluded cohort (no reconciliation manifest). The behavioral source dir + manifest default to the OAK raw path + the vendored manifest; override with --behavioral-dir / --manifest.

4d. Make the staged tree a DataLad dataset (datalad)

Version-control the staged BIDS tree with DataLad so large NIfTIs are git-annex'd while text sidecars (.tsv/.json) stay in plain git (text2git):

fw2bids datalad $SCRATCH/bids_staging/discovery

Idempotent — on an already-created dataset it just datalad saves (picking up new/changed files). It shells out to DataLad 1.6, which requires git-annex ≥ 10 — Sherlock's host git-annex is 8, so this stage must run through the container (it bakes git-annex 10). Submit it via the --container submit layer, which renders + queues the Slurm job for you:

fw2bids submit datalad --cohort discovery --container --staging $SCRATCH/bids_staging

(The bare fw2bids datalad <path> form runs the stage in-process and only works in an environment that already has git-annex ≥ 10 — i.e. inside the container.)

4e. Render the data-selection channels (select)

The terminal stage renders the three data-selection channels into the DataLad-tracked tree by shelling network_qa (selection logic lives there; network_fmri only orchestrates it). It runs network-qa compile → .bidsignore + scans.tsv + per-pipeline bids-filter_<pipeline>.json, then datalad saves them. Like datalad it needs git-annex ≥ 10, so it must run through the container:

fw2bids submit select --cohort discovery --container --staging $SCRATCH/bids_staging

Two passes. This is pass 1: it compiles only the fMRIPrep-independent generators — short_run (aborted/short scans) + behavioral (missing / non-monotonic events). The motion + lev1_outlier generators need fMRIPrep confounds / lev1 QC and fold in later during the network_glm / QA phase (pass 2), re-rendering the same channels. Which tasks + canonical anat acquisition each pipeline's bids-filter uses is declared in config/selection.json. See docs/DATA-SELECTION.md for the full rationale.

Skipped for the excluded cohort (no selection layer), exactly like events.

Whole pipeline in one command (pipeline)

Rather than submitting each stage by hand, fw2bids pipeline chains the full DAG (curate → export → merge → trim → events → datalad → select) with --dependency=afterok wiring, so each stage starts only after the previous one succeeds:

fw2bids pipeline --cohort discovery --staging $SCRATCH/bids_staging
fw2bids pipeline --cohort discovery --staging $SCRATCH/bids_staging --dry-run   # inspect first

events and select are skipped automatically for the excluded cohort.

After select completes, promote the staged tree to its canonical Oak location and set it read-only, so the reviewed selection channels are the frozen record.