stuntd 0.1.3

What's new

Novelty gate. A head trained with 0.1.3 keeps the encoder vectors of its training rows. A request unlike anything it trained on goes to the provider with reason=novel, however confident the head looks. On the support demo with one template per category left out of training, the gate sent 100% of the tickets from the unseen templates to the provider; without it, all three heads were sure on 40% of them and all three right on only 61% of those. "What's the weather in Paris", asdf qwer zxcv and an empty state were stopped too; without the gate the heads answered the weather question at confidence 1.00. The cost: about 6 points fewer requests answered locally on familiar tickets (76.6% to 70.6%), with accuracy unchanged. It is on by default; training.novelty_quantile and serving.novelty_gate change that.

An auto loop you can leave running. auto_retrain counts distinct texts, not repeated captures, and waits training.auto_retrain_min_minutes (30) between runs of a site. In a decision with several fields, a field that is live keeps being compared with the provider while the others are still in shadow, so it can still be demoted. A retrained head that replaced a live one shows as shadow (retrained, was live) instead of switching back silently. stuntd enable PARENT enables every field of a decision.

Honest local mode. Without a Jev provider, a request under the threshold is answered zero-shot by the base checkpoint. serving.local_fallback = "head" answers it with the head instead, report --gold now scores what is actually served under that setting, and enable warns when a head would answer less than half of the traffic.

Honest report. Holdout agreement and the operating point come with a 95% interval and the rows behind them, with a warning when there are too few. "Surest mistakes" show the probability the head gave and whether each one would have been served.

Sites. stuntd import skips rows already recorded (--dry-run to see the counts). stuntd site rm and stuntd site rename exist; a renamed site keeps receiving the requests of its old name. status shows the schema name next to a hash site and groups fields under their decision.

Operations. GET /healthz, stuntd stop, one progress line per epoch during training, a startup line that says what is served, reason=unsupported-schema for schemas stuntd cannot learn, and pip install "stuntd[train,jev]" for the local Jev quickstart.

Behaviour changes to know

• The novelty gate is on for heads trained with 0.1.3; heads from earlier versions serve exactly as before.
• auto_retrain retrains less often on traffic with many repeated requests.
• Importing the same file twice no longer records its rows twice.

Thanks

To Dipankar Sarkar, whose review of the published support heads on Hugging Face showed that the demo's test leaked templates and that a head's confidence says nothing about inputs it never saw, and to the people who tried 0.1.2 and said where it got in the way.