v0.8.2 — Fix: NaN in plot_state_feature_clustermap

Resolves the follow-up flagged in v0.8.1: plot_state_feature_clustermap
no longer crashes on per-Leiden-cluster mean matrices that contain NaN
entries. The fix is designed to preserve as much data as possible — drop
only what is unrecoverable, keep partial-signal columns, and render the
genuinely unknown cells as gray so the viewer can tell observation apart
from imputation.

Fixed

• plot_state_feature_clustermap accepts a new nan_policy kwarg.
  Defaults to "mask_impute": drop columns with no signal anywhere
  (all-NaN across every Leiden cluster), impute remaining NaN cells with
  the column median of finite values, and render them as gray via
  seaborn.clustermap's mask= argument. Alternatives: "drop" (drop
  any column with one or more NaN cells; no imputation), "raise" (loud
  error listing NaN-bearing statistic names for diagnostics).
• _collapse_repeated_measures now uses np.nanmean. Previously a
  single NaN in any window-repeat of a statistic propagated NaN through
  the entire post-collapse cell; now it reflects the windows where the
  statistic was observed.
• Single-column edge case handled. When nan_policy="drop" reduces
  the kept-statistic count to 1, the column dendrogram is skipped
  (col_cluster=False) so seaborn's internal scipy.pdist doesn't fail
  with "empty distance matrix".
• MCP tool surfaces the policy and diagnostics. The tool's summary
  dict gains nan_policy: str, dropped_statistics: list[str], and
  imputed_cell_count: int so the calling agent can see what the viz
  decision did.

Hardened (defensive; no observable change for current pipelines)

• cluster_ensemble switches .mean() -> np.nanmean() for the
  per-Leiden-cluster mean computation, and drops all-NaN columns of
  leiden_cluster_means before scipy.spatial.distance.pdist to keep
  the Ward linkage finite. In the current code path, sklearn's
  kneighbors_graph already rejects NaN-bearing embeddings before this
  code runs, so these changes are no-ops; they harden future paths
  (alternative clustering strategies, manually constructed cluster Zarrs
  carrying NaN) without altering existing behaviour. Existing
  tests/unit/test_cluster_leiden_ward.py tests pass unchanged.

NaN semantics (the rationale)

In every observed source, NaN in this pipeline means the statistic
could not be computed because the underlying cells / interactions were
not present — a structural absence, not a low-sample noise estimate.
mean_nearest_neighbor_distance_by_type returns NaN when mask.sum() < 2
(can't compute pairwise NN with fewer than 2 cells). cell_type_ratio
returns NaN when both numerator and denominator types are absent.
marker_statistics_by_type produces NaN when the type's population is 0.
The summarize/aggregate.py NaN-fill at line 107 marks the gap when a
timepoint's panel didn't emit a statistic because the underlying cells
weren't there.

That semantic rules out zero-fill (claiming "neighbour distance is 0 µm"
or "ratio is 0" when the cells don't exist is wrong) and any imputation
that hides the absence. The numerical value placed in a masked cell is a
computational sentinel needed only because seaborn cannot accept NaN;
the gray mask is the truth-telling layer.

Unchanged (by design)

• summarize_ensemble still emits NaN where a statistic can't be
  computed (schema stability per ADR 0009 + intentional aggregator
  NaN-fill).
• normalize_ensemble's fill_nan_with default of 0.0 still handles
  fully-NaN columns. Operators who want NaN preserved through normalize
  set it to NaN explicitly.
• tmelandscape.viz.* stays backend-agnostic (per v0.8.1).

Verification snapshot

• uv run pytest tests/unit tests/integration -q — 506 passed (was
  500; 5 new viz NaN-policy tests + 1 new integration test).
• uv run ruff check . / uv run ruff format --check . /
  uv run mypy src — clean.
• Manual end-to-end against M00_S01 canary cluster zarr:
  plot_state_feature_clustermap produces a PNG with imputed cells
  rendered gray; the MCP tool summary lists the dropped all-NaN
  statistics and the imputed cell count.

v0.8.1 — Fix: MCP plot tools on macOS (Agg backend)

Fixed

• MCP plot tools no longer crash on macOS. tmelandscape-mcp now
  forces the Agg backend before any matplotlib import in
  tmelandscape.mcp.server, so the 11 figure-producing tools (plot_*)
  succeed when FastMCP dispatches them on worker threads. Previously, the
  default MacOSX GUI backend required the main thread and raised
  "Cannot create a GUI FigureManager outside the main thread..." on the
  first plt.figure() call. Reproduced 2026-06-04 against v0.8.0 on
  macOS 14.x while running the M00_S01_canary pipeline.

Unchanged (by design)

• tmelandscape.viz.* modules contain no matplotlib.use(...) calls.
  Direct Python callers (Jupyter, scripts) keep whatever backend they
  chose. The fix is isolated to the MCP server entry.
• Operator override is honoured: launching with
  MPLBACKEND=pdf tmelandscape-mcp writes PDFs instead of PNGs, since
  the server uses os.environ.setdefault(...) not a hard assignment.

Verification snapshot

• uv run pytest tests/unit tests/integration -q — 500 passed (was 499;
  the new test_mcp_server_forces_agg_backend adds one).
• uv run ruff check . / uv run ruff format --check . /
  uv run mypy src — clean.
• python -c "from tmelandscape.mcp.server import mcp; import matplotlib; print(matplotlib.get_backend())" — prints agg.

Known follow-ups (not addressed here)

• plot_state_feature_clustermap still fails on Leiden cluster-mean
  matrices that contain NaN entries (common when *_by_type spatial
  statistics are computed on cell types that start at population 0 and
  arise via transformations — e.g. M00's CD8 / M1 / M2 / exhausted-T).
  Open follow-up: decide whether the figure function skips NaN rows /
  columns automatically, emits a more actionable error, or whether
  summarize_ensemble drops all-NaN stats upstream. Flagged by the
  M00_S01_canary report; tracked separately.

v0.8.0 — Graph-coloring IC generation (breaking)

Step 1 (generate_sweep) now produces per-replicate initial-condition CSVs
that match a user-supplied typed source tissue in both per-type proportions
and per-type-pair edge counts. The previous wrapper was a single-type
plumbing stub built on tissue_simulator.ReplicateGenerator with a hardcoded
400×400×20 µm tissue and one generic "cell" type — it ran no
simulated-annealing type assignment and silently produced uniform-random
output bearing no relation to the model's cell types or spatial structure.
The rewrite uses the correct entry points (load_tissue_from_csv,
SpatialNetworkAnalyzer, GraphColorizer.colorize) and threads scaffold
strategy, SA schedule, and graph mode through SweepConfig.

This is a breaking change under pre-1.0 SemVer: old SweepConfig JSONs
and old SweepManifest JSONs do not validate (the new fields are
required). There is no migration shim. See "Migration" below.

Added

• SweepConfig.ic_source (required) — discriminated union of
  IcSourceOwnData (mode A: user's typed CSV) and IcSourceStructured
  (mode B: agent-described tissue realized to a typed CSV upstream).
  Both modes end in a single typed-coordinate CSV; the mode field exists
  for manifest provenance. The upstream /tme-landscape skill flow is
  responsible for producing the structured CSV.
• SweepConfig.ic_scaffold_strategy — "uniform_random" (default) or
  "perturb_source". The latter copies source positions and jitters by
  Normal(0, 0.5 * min_radius); useful for preserving source morphology.
• SweepConfig.ic_sa_params (SAParams model) — initial_temp=10.0,
  final_temp=0.01, cooling_rate=0.9997, max_iterations=60_000.
  Tuned empirically against ~250–2000 node TME graphs.
• SweepConfig.ic_network_mode (default "radius") and
  ic_network_radius_um (default None ⇒ 2.5 * max(source cell radius)
  at call time).
• SweepRow.ic_scaffold_seed — deterministic per-replicate seed
  (spawned from SweepConfig.seed via numpy.random.SeedSequence).
• SweepRow.ic_sha256 — SHA-256 hex digest of the IC CSV bytes.
  Lets the downstream content-addressing layer in llm_abm_landscapes
  verify byte-level reproducibility without a post-hoc walk of the
  manifest's IC directory.
• SweepRow.ic_sa_final_cost — diagnostic; cost of the SA solution
  on the matched-N scaffold graph.
• SweepRow.ic_achieved_proportions — per-type proportion realized
  in the generated IC.
• IcGenResult dataclass — return type of
  generate_initial_conditions, carrying the five diagnostics above per
  replicate so the orchestrator does not re-read the CSV.
• Parquet sidecar columns — ic_scaffold_seed: int64,
  ic_sha256: string, ic_sa_final_cost: float64, and a JSON-encoded
  ic_achieved_proportions_json: string (kept as JSON so the schema stays
  stable across sweeps with different cell-type vocabularies).
• Static test fixture tests/data/ic_source_structured.csv — a
  30-cell hand-rolled tumor-disc / fibroblast-ring / CD8-annulus tissue
  with CD8↔tumor edges = 0 under the default radius graph; used by every
  tests/unit/test_sampling_tissue_init.py test.
• tests/integration/conftest.py — session-scoped fixture
  structured_source_csv that builds a ~30-cell ringed source tissue via
  tissue_simulator.TissueSection.generate_cells at a fixed seed.

Changed

• tissue_simulator dependency pinned to v0.1.9 (was v0.1.4).
  v0.1.9 ships explicit seed= on TissueSection / SpherePacker /
  GraphColorizer, so the previous
  tissue_simulator.{packing,tissue}.np.random.default_rng monkey-patch
  is gone. v0.1.9 also exposes GraphColorizer, SpatialNetworkAnalyzer,
  and load_tissue_from_csv — the symbols this rewrite needs.
• generate_initial_conditions signature rewritten:
  • Added source_csv, scaffold_strategy, sa_params,
    network_mode, network_radius_um.
  • Removed target_n_cells, cell_radii_um, tissue_dims_um,
    similarity_tolerance (all physical parameters now come from the
    source CSV per acceptance criterion #4 of the spec).
  • Return type changed from list[Path] to list[IcGenResult].

Removed (breaking)

• generate_sweep kwargs target_n_cells, cell_radii_um,
  tissue_dims_um, similarity_tolerance — geometry comes from
  config.ic_source.source_csv now.
• MCP generate_sweep_tool kwargs target_n_cells,
  similarity_tolerance.
• CLI tmelandscape sample flags --target-n-cells and
  --similarity-tolerance. The CLI now exposes only --manifest-out
  and --ic-dir; all IC knobs live on the JSON config.
• _seeded_default_rng_factory and the ExitStack /
  patch.object block in tissue_init.py — no longer needed now that
  upstream honours explicit seeds.

Migration

Adding ic_source (required) to every SweepConfig JSON is the minimum
to load an existing config under v0.8.0. Both shapes are valid:

{
  "mode": "own_data",
  "source_csv": "/abs/path/to/your_tissue.csv"
}

{
  "mode": "structured",
  "source_csv": "/abs/path/to/agent_realised_tissue.csv",
  "description": "tumor disc + fibroblast ring + CD8 annulus"
}

Old SweepManifest JSONs cannot be loaded; they lack the four new
SweepRow fields. Regenerate them by re-running step 1 against the same
config (with ic_source added).

Verification snapshot

• uv run pytest tests/unit tests/integration -q — 499 passed
  (459 unit + 40 integration).
• End-to-end smoke on the static fixture: per-replicate proportions match
  source exactly (10 tumor / 12 fibroblast / 8 CD8 → 10/12/8); per-IC
  CD8↔tumor edges = 0 under scaffold_strategy="perturb_source".
• Same (config, seed, source_csv) produces byte-identical CSVs (same
  ic_sha256) — verified against v0.1.9 GraphColorizer.seed=.

Acceptance criteria coverage (from spec)

1. Multi-type structured source ⇒ matched proportions and preserved
   CD8↔tumor exclusion: ✅ (test_proportions_match_source_exactly,
   test_cd8_tumor_edge_fraction_preserved_with_perturb_source).
2. Reproducibility: ✅ (test_same_seed_yields_identical_csvs).
3. Manifest carries ic_sha256 per row: ✅ (parquet test +
   integration test).
4. No hardcoded cell type or default dims: ✅ (geometry from source CSV).
5. MCP generate_sweep round-trips new config fields: ✅
   (test_mcp_tool_matches_python_api).
6. Existing tests pass with extensions for the new fields: ✅
   (test_config_sweep.py round-trip + discriminator dispatch).

Out of scope (per spec, restated)

• Generation of the structured source CSV upstream (skill flow +
  PhysiCell place_initial_cells).
• /tme-landscape skill prompts for ic_source mode and source path.
• Content-addressing layer (model_hash / sweep_hash / per-IC
  sha256 binding) — lives in llm_abm_landscapes. This release just
  emits ic_sha256 per row.