-
Notifications
You must be signed in to change notification settings - Fork 0
Believability Testing
The engine is validated two ways: a fast deterministic sanity gate (does every run stay mathematically sane?) and a blind LLM judge (does each run read like a believable stretch of a life, in character?).
-
700 one-day scenarios + 700 multi-day scenarios, seeded and deterministic. Each is a timeline
of forcing events done to one NPC — vocabulary:
insult,command,food_given,help,weather. The NPC runs through a closed seeking loop against a mock world at a believable timescale (a 24 h day; sleeps at night, wakes recovered). - Every scenario is run twice — with the outburst overlay OFF and ON — giving 2,800 tests.
Checks applied to every tick of a run (no LLM): states clamped to range and never NaN; hunger sane (not starving in minutes, meals actually feed, never pegged); a sleep stretch each night; the fast temper drains overnight; stress not pegged near 1.0; calm by most dawns; not frozen in a single mode.
Records are batched 10 at a time (same persona). One fresh agent judges each batch with no answer key and no comparison between records — it sees only the persona profile, the observable narration, and a neutral rubric:
- Sane as a day / days in a life — believable pacing, sleeps and recovers, nothing absurd.
- Not wrong for this person — nothing contradicts the persona's profile.
A record PASSES if both hold, else it is FLAGGED with a one-line reason.
2,618 / 2,800 PASS — 93.5%.
| sub-corpus | PASS | rate |
|---|---|---|
| Day · burst OFF | 650/700 | 92.9% |
| Day · burst ON | 655/700 | 93.6% |
| Multiday · burst OFF | 661/700 | 94.4% |
| Multiday · burst ON | 652/700 | 93.1% |
| overall | 2618/2800 | 93.5% |
Per persona: wojslaw 96.2%, edda 95.5%, cichy 95.2%, welf 93.5%, branic 92.8%, halgrim 91.5%, lutek 89.8%.
Signal: in this run the outburst overlay is believability-neutral (burst OFF 93.6% vs ON 93.4% —
1311/1400 vs 1307/1400), so the remaining flags are not burst-driven. The recurring flag families are
(a) a kindness met with non-warmth or even a snap (expression mislabel / displaced discharge), and
(b) non-compliance toward a respected source (relational gating) — both calibration targets rather
than structural faults. Full triage: eval/hourly_runs/TRIAGE.md.
The Moral Tension overlay (guilt/secrets/lies/accusation/suspicion/betrayal) is opt-in and
judged on its own. Because moral plausibility is not a binary "sane day" question, its corpus is scored on
a fuzzy 1–5 quality scale (believability + curve_plausibility) rather than PASS/FAIL, after a
deterministic pre-filter. Post-calibration it scores 4.0–4.83/5; the serious-guilt (~72h) and
suspicion (24h) half-lives are judge-validated. Crucially, the overlay is byte-identical to a non-moral
build on the base corpus, so the PASS numbers above are unchanged by it. Full detail:
eval/MORAL_REPORT.md.
-
Summary:
eval/hourly_runs/FINAL_report.md -
Every test (stimulus → observed behaviour → verdict):
eval/hourly_runs/TEST_REPORT.md -
Generators:
eval/hourly_report.py(summary),eval/test_report.py(per-test) — pure read/render, reproducible.