-
Notifications
You must be signed in to change notification settings - Fork 0
Believability Testing
The engine is validated two ways: a fast deterministic sanity gate (does every run stay mathematically sane?) and a blind LLM judge (does each run read like a believable stretch of a life, in character?).
-
700 one-day scenarios + 700 multi-day scenarios, seeded and deterministic. Each is a timeline
of forcing events done to one NPC — vocabulary:
insult,command,food_given,help,weather. The NPC runs through a closed seeking loop against a mock world at a believable timescale (a 24 h day; sleeps at night, wakes recovered). - Every scenario is run twice — with the outburst overlay OFF and ON — giving 2,800 tests.
Checks applied to every tick of a run (no LLM): states clamped to range and never NaN; hunger sane (not starving in minutes, meals actually feed, never pegged); a sleep stretch each night; the fast temper drains overnight; stress not pegged near 1.0; calm by most dawns; not frozen in a single mode.
Records are batched 10 at a time (same persona). One fresh agent judges each batch with no answer key and no comparison between records — it sees only the persona profile, the observable narration, and a neutral rubric:
- Sane as a day / days in a life — believable pacing, sleeps and recovers, nothing absurd.
- Not wrong for this person — nothing contradicts the persona's profile.
A record PASSES if both hold, else it is FLAGGED with a one-line reason.
2,731 / 2,799 PASS — 97.6%.
| sub-corpus | PASS | rate |
|---|---|---|
| Day · burst OFF | 694/700 | 99.1% |
| Day · burst ON | 684/700 | 97.7% |
| Multiday · burst OFF | 689/700 | 98.4% |
| Multiday · burst ON | 664/699 | 95.0% |
| overall | 2731/2799 | 97.6% |
Per persona: edda 100%, wojslaw 99.0%, cichy 98.8%, lutek 97.8%, welf 96.8%, halgrim 96.2%, branic 94.5%.
Signal: the outburst overlay measurably lowers believability (burst OFF 98.8% vs ON 96.4%), concentrated in multiday burst-ON. The recurring flag — "erupts at a kindness with no provocation" — points at the displaced-discharge edge as a calibration target (the vent should be a stability valve for coinciding pressures, not something that snaps at a kind act).
The Moral Tension overlay (guilt/secrets/lies/accusation/suspicion/betrayal) is opt-in and
judged on its own. Because moral plausibility is not a binary "sane day" question, its corpus is scored on
a fuzzy 1–5 quality scale (believability + curve_plausibility) rather than PASS/FAIL, after a
deterministic pre-filter. Post-calibration it scores 4.0–4.83/5; the serious-guilt (~72h) and
suspicion (24h) half-lives are judge-validated. Crucially, the overlay is byte-identical to a non-moral
build on the base corpus, so the PASS numbers above are unchanged by it. Full detail:
eval/MORAL_REPORT.md.
-
Summary:
eval/hourly_runs/FINAL_report.md -
Every test (stimulus → observed behaviour → verdict):
eval/hourly_runs/TEST_REPORT.md -
Generators:
eval/hourly_report.py(summary),eval/test_report.py(per-test) — pure read/render, reproducible.