v0.1.0: monorepo merge — unified training + inference, canonical CellTypeAnnotator, archive-free inference, vendored baselines

[xuefei-wang]

Summary

This is the v0.1.0 release cut. It merges the separate training repository
(deepcelltypes-cell-type-assignment-pytorch) into this repo and replaces the
legacy CellTypeCLIPModel inference path with the current canonical
CellTypeAnnotator model.

Before this PR, vanvalenlab/deepcell-types was inference-only — it shipped
CellTypeCLIPModel, the dct_kit/ helpers, and a top-level __init__ that
exported just predict. After it, a single package covers training and
inference: inference stays a plain pip install deepcell-types, the full
training pipeline lives behind a [train] extra, and the four paper comparison
baselines are vendored behind per-baseline extras.

⚠️ This PR contains breaking changes — see the section at the bottom.

The CHANGELOG.md
0.1.0 entry is the authoritative, line-by-line record; this description
summarizes it.

Canonical model

model.py is rewritten around CellTypeAnnotator; CellTypeCLIPModel /
CellTypeDataEncoder are removed (no shim). Canonical training defaults
(scripts/train.py, click-based CLI): --resnet_channels 48,
--domain_weight 0.1, --best_metric macro_f1, --lr 1e-3, and the new
--ct_head_arch resmlp.

• Residual-MLP cell-type head is now the default (ct_head_arch="resmlp",
  width-512 / depth-4) for fresh training, replacing the legacy 3-layer MLP.
  Trained on the frozen backbone via scripts/retrain_head.py (the two-stage
  sampler-off recipe). Checkpoints record config["ct_head_arch"] and
  predict() auto-detects the head from the state dict, so the released
  (legacy-MLP) checkpoint still loads unchanged. resmlp and mlp
  checkpoints have different head shapes and are not interchangeable.
• Mean-intensity injection — per-cell mean marker intensity is scattered
  into a marker-position vector and injected as a CLS residual. The output
  projection is zero-init, so warm-starting from a checkpoint preserves
  predictions at step 0.
• DANN domain adaptation via a gradient-reversal head, on by default
  (--domain_weight 0.1; 0 disables it).
• Adapter-style fine-tuning: --freeze_backbone trains only the
  mean-intensity branches on top of an existing checkpoint; --unfreeze_ct_head
  additionally co-adapts the CT head / CLS token / final norm without unfreezing
  the transformer backbone.
• Padding-channel positions are explicitly zeroed (out-of-place masked_fill,
  AMP-safe) through the channel encoder, fusion, and mean-intensity paths so
  masked tokens contribute exactly zero rather than leaking
  proj.bias/spatial_feat into the transformer.
• Self-describing checkpoints: scripts/train.py bundles ct2idx,
  n_heads, and compat_marker0_zero, and inference asserts the vocabulary
  ordering matches (a permuted vocabulary previously passed the count-only
  check and silently mislabeled cells). --resume_path validates n_heads /
  n_celltypes / resnet_channels / d_model before restoring optimizer
  state.

Canonical-only inference

• Archive-free by default: the marker / cell-type registry ships as a small
  packaged vocab.json snapshot, so pip install deepcell-types +
  download_model() is enough to run predict() — the multi-GB TissueNet zarr
  archive is no longer required (pass zarr_path= / set
  DEEPCELL_TYPES_ZARR_PATH only if you need the tissue→cell-type mapping).
  Verified identical predictions with vs. without the archive on the paper
  checkpoint.
• Post-hoc abstention is opt-in. predict(ct_abstention_k=...) defaults to
  None, returning the raw argmax label for every cell. Set a float and cells
  below an IQR fence on the per-FOV confidence distribution are relabeled to the
  "Unknown" sentinel; ct_abstention_k=0.2 reproduces the paper headline
  operating point. The pre-abstention argmax is preserved in cell_types_raw
  when return_probabilities=True. (Default is None, not on-by-default, so
  the plain list[str] return is never silently rewritten at a benchmark-tuned
  threshold.)
• Custom preprocessing hook: predict(..., preprocess=...) overrides the
  per-FOV normalization without retraining, backed by a bounded op library
  (apply_config, make_preprocessor, DEFAULT_CONFIG) and a
  composition-guided adaptation loop (skills/preproc-adapt/).
  make_preprocessor(DEFAULT_CONFIG) reproduces the default bit-for-bit.
• Input-hardening / training-parity fixes: all-zero marker channels are now
  masked at inference (matching the training dataloader — an all-zero channel
  was previously fed as a present zero token); duplicate channels resolving to
  the same canonical marker are de-duplicated; non-finite (NaN/inf) raw
  now raises a clear ValueError instead of a poisoned softmax; per-cell
  tensors are sized to the channels actually present rather than the global
  MAX_NUM_CHANNELS. The first three can change predictions for affected FOVs.
• The bright-spot clip percentile (DCTConfig.PERCENTILE_THRESHOLD) is now
  99.9, matching the recipe the training archive was built with (was 99.0).
  Shifts ~5% of predicted labels; reproduces the canonical predictions slightly
  better on a held-out test sample (92.5% vs 91.9% argmax agreement).
• predict(return_probabilities=True) returns a PredictionResult dataclass
  with the full per-cell softmax matrix, cell indices, and the pre-abstention
  argmax labels (cell_types_raw).
• Loading uses weights_only=True where possible; a missing checkpoint raises a
  clear FileNotFoundError pointing at download_model().

New public API

predict, DCTConfig, PredictionResult, preprocess_fov, apply_config,
make_preprocessor, and DEFAULT_CONFIG are importable from deepcell_types
directly. preprocess_fov(raw, mask, native_mpp, channel_names) → PreprocessedFov is the standalone preprocessing entry point.

Monorepo: training pipeline

• deepcell_types.training ships behind pip install "deepcell-types[train]":
  config.py, dataset.py, archive.py, annotations.py,
  baseline_features.py, gold_metadata.py, losses.py, metrics.py,
  patch.py, utils.py, abstention.py.
• Scripts under scripts/: train.py, pretrain.py, predict.py,
  retrain_head.py, generate_openai_embeddings.py, generate_splits.py,
  split_val_for_test.py, plus the release-archive gate
  (validate_archive_contract.py, check_release_archive.sh).
• Canonical split manifests committed under splits/, so the published
  train/val/test partition is reproducible from the repo. Baselines and the main
  model select checkpoints against one shared canonical val split via
  --val_split_file (200k-cell cap, seed 42).
• Experiment logging is plain Python logging — no Weights & Biases anywhere
  (--enable_wandb is gone; confusion matrices save locally as PNGs).
• zarr>=3.1 pulls the Python floor up to 3.11 for the train extra.

Baselines

• Four paper comparison baselines vendored under deepcell_types/baselines/
  (cellsighter, maps, nimbus, xgb), invoked through the unified runner
  python -m deepcell_types.baselines <name>, each with a self-contained
  install extra (baseline-cellsighter, baseline-maps, baseline-nimbus,
  baseline-xgboost).
• All baselines unified onto the DCT sampler by default (--class_balance
  defaults to dct/sqrt), so cross-method comparisons isolate the modeling
  choice rather than the balancing scheme. Each baseline's native sampler stays
  available via --class_balance and is reported separately in the appendix.
• extract_features_from_zarr(missing_value=...) lets each baseline choose its
  absent-marker sentinel: MAPS / CellSighter keep 0.0; XGBoost passes
  np.nan so absent markers route through XGBoost's learned missing
  direction. The feature matrix records a present_markers mask and the cache
  stays missing-value-agnostic.
• Each baseline ships a README documenting every deviation from its upstream
  source; third-party licenses are tracked in deepcell_types/baselines/NOTICE.

Breaking changes

• CellTypeCLIPModel removed. No shim — use from deepcell_types import predict, DCTConfig.
• All predict() arguments after mpp are keyword-only, preventing
  accidental transposition of the adjacent string arguments. device= is the
  preferred spelling (device_num= remains a deprecated alias).
• predict(num_workers=...) default is now 0 (was 24) — 24 workers
  OOM'd machines with <64 GB RAM.
• Clip percentile 99.0 → 99.9 shifts ~5% of predicted labels.
• Baselines default to the DCT sampler (--class_balance dct) instead of
  each method's native class-balancing scheme.
• Backbone training no longer applies per-class FocalLoss weights — the
  WeightedRandomSampler is the sole rare-class balancer. --no_class_weights
  and compute_class_weights are removed; the no-flags default now reproduces
  the released-checkpoint recipe.
• from deepcell_types.predict import PredLogger is gone (now the private
  _InferenceResultBuffer).

Packaging / infra

• Package data now ships vocab.json, channel_mapping.yaml, and
  training/config/*.yaml (incl. combined_celltypes.yaml), which were
  previously outside the package tree and absent after pip install.
• NumPy 2.0 compatibility in the inference path (np.ptp free function
  removed in NumPy 2.0 → max - min), so a fresh pip install runs.
• tifffile declared in the [train] extra; openai declared and pinned.
• CI workflow added (.github/workflows/ci.yml); inference vs. [train] test
  boundary enforced.
• LICENSE matches the lab's Modified Apache 2.0 template; NOTICE aligned to
  the vanvalenlab convention and attributes vendored third-party code.

Tests

35+ test modules under tests/ (plus tests/baselines/) covering canonical
inference, abstention CLI, checkpoint round-trip, dataset/split/sampler
behavior, preprocessing + the preprocess hook, losses, hierarchical eval,
archive-contract validation, baseline feature splits, and vendored-baseline
equivalence against upstream.