feat(simulator): persona-drift metric + drift_alert stream events (#29)

[flyworker]

Closes #29. Productizes Kenneth Li's persona-drift methodology (ArXiv 2402.10962) as a runtime metric.

The framing decision that shaped everything

The archetype constrains the simulator, not the target agent — so drift scores the synthetic customer's turns. It is a validity check on the evaluation itself ("was this run still testing the persona you authored?"), not a quality score for the agent under test. Reading result.drift as an agent metric would be a confident misuse, so that framing is stated in the module docstring, the types, and the docs.

Scoring

Four signals per probe point — character_break (1.0), disposition_violation (0.9), advisory_inversion (0.6), format_break (0.2) — combined noisy-OR per turn, since they're independent evidence for one latent event.

Aggregate is 0.5·max + 0.5·mean. A plain mean is actively wrong here: one total break in a 20-probe run averages to 0.05 and never alerts, which is exactly the failure #29 describes. The max term gives a statable guarantee — any single complete character break scores ≥ 0.5 regardless of run length — while the mean still rewards recovery and punishes persistent low-grade drift. Both are asserted in tests.

disposition_violation is direction-sensitive, which is the part a reimplementation would most likely get wrong: a naive "leakage = drift" scorer fires on every well-run simulation, because eliciting hidden facts is the agent's job. cautious counts disclosure as drift only when volunteered; guarded requires prompting plus established trust; open treats disclosure as never drift and evasion as the violation. One test runs the same transcript against two dispositions and asserts opposite verdicts.

Calibration was empirical, not by review

An adversarial in-character corpus caught three markers shipped on plausibility that were false positives on ordinary customer speech — "the simulation" (as in "the simulation results from the calculator", scoring a maximum 1.0), "my instructions say", and "i'm an assistant" (matching "I'm an assistant manager at a retail store"). All removed, with the rejects documented in-file so nobody re-adds them.

Independent verification

I scored a corpus I wrote myself, separate from the repo's fixtures, including those three false-positive sentences:

in-character corpus : 0.000   alert=False
broken corpus       : 0.633   alert=True

Also confirmed the unmeasured-vs-clean distinction is real rather than cosmetic:

1-turn break : score=0.0  measured=False  reason='insufficient_persona_turns'
4-turn breaks: score=0.633 measured=True

That matters because 0.0 = perfect retention and 0.0 = not measured are different claims, and a partner averaging drift.score across a corpus would otherwise silently pull the mean toward zero.

Backward compatibility (the package's public API has been frozen since v1.0.0): SimulationResult.drift and TurnEvent.drift both default to None, so existing callers and stored results are unaffected.

Tests: 238 passed, 4 skipped — 50 new against a 188 baseline.

Two deliberate deviations from the acceptance criteria

"≥4 turns" was under-inclusive, so every call gets a DriftMetric; runs too short to measure carry measured == False and a reason rather than a misleading 0.0.

Probes are not auto-generated from archetype.description, on purpose: a probe that restates the persona re-primes the model's attention on it — the exact quantity being measured — so description-derived probes would partially erase the drift they exist to expose. Built-in probes are persona-agnostic; authors can supply their own via drift_probes.

What it does not detect — stated in the docs, not buried

Semantic drift that preserves register (a persona still sounding like a customer while contradicting its own biography) is the big one and needs a model-based judge. Also paraphrased leakage, tone/stance drift, non-English transcripts (marker sets are English-only, so the shipped Punjabi-register archetype under-scores on its Punjabi turns), and turns the cadence didn't sample. A model-graded method can register under a new method= identifier without touching a signature.

One thing to decide before merging

drift_alert is a new TurnEvent.kind emitted by default. Consumers ignoring unknown kinds are fine and it only fires on drifted runs, but code asserting an exhaustive kind set would break. Default-on was judged correct (#29's premise is that partners are running blind), with DriftOptions(alert=False) plus an upgrade note — flipping it to opt-in is a one-line change if you'd rather.

Also worth noting: this adds a new public surface to a package whose API has been frozen since v1.0.0, so it wants a release-notes entry before the next cut.