Skip to content

v0.8.1 - Statistical correctness and security hardening

Choose a tag to compare

@strot-ai strot-ai released this 13 Oct 16:07
· 16 commits to main since this release

Correctness release, prompted by an external technical review of v0.8.0 - with thanks to the reviewer.

Statistical correctness (Priority 1)

  • Fold-split calibration: temperature is now fitted on one half of the teacher-labeled points and the conformal quantile on a disjoint half, preserving split-conformal validity (fitting both on the same points made the scores non-exchangeable with fresh records).
  • Corrected guarantee wording everywhere: split conformal prediction targets marginal coverage of at least 1-alpha under exchangeability, with abstention on non-singleton prediction sets. Wrong dispatches are <= alpha of ALL records; the conditional-on-dispatch bound (alpha / dispatch rate) is now reported as conditional_error_bound_estimate in the artifact's CALIBRATION metadata.
  • Empirical coverage test added: fitted on 300 synthetic points, the measured wrong-dispatch rate on 2,000 held-out points stays within alpha.

Security (Priority 3)

  • The lint on generated code now rejects stdlib escape hatches (os, sys, subprocess, socket, urllib, pickle, ctypes, threading, ...) and dynamic-execution calls (eval, exec, open, import, compile, input), with repair-loop feedback.
  • Documented plainly: this is validation, not a sandbox - compiling untrusted intent requires external isolation.

Reproducibility (Priority 2)

  • New benchmarks/ directory with the exact 398-record routing benchmark, the input-formatting ablation, and methodology notes (the compiler never sees ground-truth labels; the 100% figure is a metric-tuned ceiling; compile variance is documented).

Positioning

STROT - the open decision compiler.

89 tests pass.