Skip to content

verify_physics_fn should include biological-realism checks per model (not just state-range + finiteness) #6

@ajaytalati

Description

@ajaytalati

Motivation

Currently every model's `verify_physics_fn` only checks STATE RANGES and FINITENESS:
```python
return {
'W_in_0_1': ..., 'Zt_in_0_A': ..., 'a_nonneg': ..., 'all_finite': ...,
}
```

This is necessary but not sufficient. A model can pass these checks while producing biologically implausible data (see #5 — SWAT Set A's Zt amplitude never reaches `c_tilde + delta_c` so deep-sleep happens only via stochastic noise floor; the simulated truth exhibits unrealistic sleep architecture).

These model-design issues should be surfaced at the model layer (this repo), not 1-2 stages downstream (psim or SMC²). Wasted GPU time on the SMC² port for SWAT was traceable to two such issues plus a framework bug.

Proposal

Extend each model's `verify_physics_fn` to also return informational biological-realism metrics. These are NOT gating (they don't fail the CLI's `PASS: physics verification` line) but they SHOULD be surfaced clearly in the run output so the modeller spots problems at simulation time.

Convention: any key ending in `_realistic` is a string `"yes"/"no"` (informational, not gating). Numeric metrics (e.g. `sleep_fraction: 0.21`) are also informational. Only `_ok` / `in*` style booleans gate.

SWAT-specific (Set A as worked example)

`models/swat/simulation.py:verify_physics`:

```python
def verify_physics(trajectory, t_grid, params):
W = trajectory[:, 0]; Zt = trajectory[:, 1]; a = trajectory[:, 2]; T = trajectory[:, 3]

# Existing range / finite checks
res = {
    'W_in_0_1':       bool((W >= 0).all() and (W <= 1).all()),
    'Zt_in_0_A':      bool((Zt >= 0).all() and (Zt <= A_SCALE).all()),
    'a_nonneg':       bool((a >= 0).all()),
    'T_nonneg':       bool((T >= 0).all()),
    'T_bounded':      bool(T.max() < 100.0),
    'all_finite':     bool(np.all(np.isfinite(trajectory))),
}

# NEW: informational realism metrics
c1 = params['c_tilde']
c2 = c1 + params['delta_c']
sleep_frac = float((Zt > c1).mean())
deep_sleep_frac = float((Zt > c2).mean())
Zt_max = float(Zt.max())

res['sleep_fraction'] = sleep_frac
res['deep_sleep_fraction'] = deep_sleep_frac
res['Zt_max'] = Zt_max
res['T_range'] = float(T.max() - T.min())
res['T_mean'] = float(T.mean())
# Realism flags (string-typed; informational only)
res['sleep_fraction_realistic'] = (
    \"yes\" if 0.25 <= sleep_frac <= 0.40 else \"no\")
res['Zt_reaches_deep_threshold_realistic'] = (
    \"yes\" if Zt_max > c2 else \"no\")
res['deep_sleep_fraction_realistic'] = (
    \"yes\" if 0.05 <= deep_sleep_frac <= 0.15 else \"no\")
return res

```

When SWAT Set A is simulated by `run_simulator.py --model models.swat.simulation.SWAT_MODEL --param-set A`, the output would now show:

```
sleep_fraction 0.057
deep_sleep_fraction 0.000
Zt_max 4.30
sleep_fraction_realistic no <- modeller spots this
Zt_reaches_deep_threshold_realistic no
deep_sleep_fraction_realistic no <- and this
all_finite PASS
PASS: physics verification
```

`PASS` still fires on the gating booleans (which are unchanged); the informational `no` flags clearly surface the realism concerns.

Other models

Each model's verify_physics_fn should be similarly extended:

  • `fsa_real_obs` / `fsa_high_res`: realistic resting-HR range, fitness-buildup cycles
  • `sleep_wake` / `sleep_wake_20p`: sleep architecture metrics
  • `bistable_controlled`: state-switching frequency

Per-model definition; `how_to_add_a_new_model/03_testing_and_docs.md` should be updated to include realism checks as part of the model-developer responsibility.

Acceptance criteria

  • Each existing model's `verify_physics_fn` includes at least 2 model-specific realism metrics + 2 `*_realistic` informational flags
  • CLI smoke test (`run_simulator.py`) still passes for each model (gating booleans unchanged)
  • `how_to_add_a_new_model/03_testing_and_docs.md` documents the realism-check convention
  • Each model's `TESTING.md` §4 (parameter sets) documents the expected realism-metric ranges per parameter set

Cross-reference

  • #5 (SWAT Set A Zt amplitude — the motivating case)
  • psim follow-ups: per-model diagnostic plot at packaging + realism-warning surfacing in validate_simrun (separate issues to be opened)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions