v0.11.1 - Remove LLM-escalation cascade

Removes the LLM-escalation cascade (escalate_to) added in v0.11.0.

Measured on tweet_eval hate, tier-2 escalation to haiku bought ~0 accuracy (64.1% vs the local model's 63.5%) at real per-record cost and a runtime LLM dependency - against STROT's principle that the runtime stays frozen, local, and free. A cascade only pays when tier-2 substantially beats tier-1, and that is a model-slot decision (a specialist or bigger model in nlu=), not a runtime feature.

What stays: target_accuracy as a LOCAL-only policy - temperature + per-class-bias corrections and a dispatch cutoff fitted at compile and baked in, no LLM at runtime. Below-cutoff records abstain; the consumer decides what happens next. This is the Jev-style contract: return the decision with its honest (calibrated) confidence and let the caller threshold - don't pay to act clever on the hard slice.

The cascade benchmark finding is kept in benchmarks/ as the record of why it was removed. 106 tests pass.

v0.11.0 - Target-accuracy policy + inference corrections + cascade

Keep optimizing until a target is met, cheaply - with inference-level corrections and a two-tier cascade, all fitted at compile, runtime frozen.

New

• target_accuracy=0.8: fit temperature + per-class bias corrections and a dispatch cutoff on held-out folds so auto-decided records meet the target; below-cutoff records abstain. Mutually exclusive with target_risk. An empirical policy, NOT a finite-sample guarantee - and it optimizes ACCURACY (see caveat below).
• escalate_to="anthropic/claude-haiku-4-5" (or openai/...): two-tier cascade. Abstaining records are arbitrated by a stronger model at runtime, bounded to registry keys (else human fallback). Only the escalated slice pays per-record cost.
• Per-class vector-scaling bias correction (corrects systematic class skew a single temperature cannot).

Validated live on tweet_eval hate - two findings worth more than the number

Full stack (distilled + target 0.8 + haiku escalation) scored 63.7% combined on 400 tweets - and taught two lessons, documented with the data in benchmarks/:

1. A cascade only pays when tier-2 substantially beats tier-1. Haiku (64.1%) essentially tied the distilled local model (63.5%) on implicit hate, so escalating 145 tweets bought ~0 accuracy at real cost. Use the cascade when the escalation arbiter genuinely dominates (Opus, a task specialist) - not when both tiers are weak on the same hard cases.
2. Accuracy-target correction trades minority recall for accuracy on imbalanced data. The fitted class bias shifted toward the majority class and hate recall collapsed to 22%. That is accuracy-optimization doing exactly what it was told - and accuracy is the wrong objective for moderation. Optimize recall/F-beta on the positive class there (a clean future lever).

The machinery is correct and cheap; it shines when tier-2 dominates and classes are balanced. On adversarial imbalanced hate detection it does not rescue the task - the bottleneck remains a decision model that can read implicit hate, which is a model-slot problem (task-trained specialist), not a policy problem.

108 tests pass.

v0.10.0 - Compile-time distillation

The compiler trains. LLM-backed fine-tuning at compile time - no RL, no runtime learning, frozen artifacts throughout.

Compile-time distillation (run(distill=True))

1. The LLM teacher labels your real sample records (chunked batches, one cheap call per 50)
2. A logistic head is trained on the frozen encoder's embeddings - pure-Python gradient descent, seconds on CPU, zero new dependencies in the core package
3. Champion rule: the distilled head ships only if it beats the zero-shot router on a held-out measurement fold; a winner that breaks golden records is rejected
4. Fold hygiene enforced: teacher-labeled records split three ways (train / measure / calibrate, deterministically shuffled). Teacher data may teach; only held-out real records judge
5. model_search scores land in artifact metadata and the episode log; distill is part of the cache key

Measured (tweet_eval hate - deliberately the hard case, full table in benchmarks/)

	zero-shot (v0.9.0)	distilled (v0.10.0)
Held-out fold	0.44	0.72 (+28pp, champion selected)
Test accuracy	47-53%	59.0% uncalibrated / 63.8% selective at alpha=0.3 (57% coverage)
Calibration at alpha=0.1	abstained on ALL (correct refusal)	abstained on ALL (a ~70% model cannot promise 10% error)

Honest limits, published with the numbers: the largest single lever measured, and still below deployability for implicit-hate moderation - the generalist encoder caps the head; task-trained specialists in the nlu= slot remain the answer for that regime. Fold scores overestimate test scores (noisy 50-point judges); report test numbers.

103 tests pass.

v0.9.0 - Measured label search

The compiler measures instead of guessing.

The v0.8.1 semantic benchmark exposed a failure mode: synthesized routing labels beat bare class names on AG News (+14pp) but LOST on Emotion (-8.3pp), because the class names were already optimal and extra wording diluted them. The compiler was guessing at wording with no way to know when to leave well enough alone.

Measured label search

compile() now scores candidate labelings - the synthesized labels AND the bare registry keys - on the teacher-labeled sample, through the actual artifact, and bakes the winner:

• Free when target_risk is set (shares the single teacher-labeling call); opt-in alone via measure_labels=True
• A winner that breaks the golden gate is rejected (goldens stay a regression gate, not an optimizer)
• Scores recorded in label_search metadata and the episode log; the flag is part of the cache key

Validated live (real model, 300 held-out test records each)

Dataset	Naive	Compiled (v0.8.1)	Measured compile (v0.9.0)
Emotion	93.3%	85.0%	93.3% - fully recovered (winner: registry keys)
AG News	72.7%	86.7%	76.7% (winner: synthesized; weaker synthesis draw)

Honest limits, documented in benchmarks/: measurement bounds the downside (floor near the best available candidate) but does not eliminate synthesis variance, and a 25-point sample is a noisy judge. Multiple synthesis candidates and larger measurement samples are the known next steps.

Also in this release: semantic-text benchmark published with all results including the losses; developer guide updated.

93 tests pass.

v0.8.1 - Statistical correctness and security hardening

Correctness release, prompted by an external technical review of v0.8.0 - with thanks to the reviewer.

Statistical correctness (Priority 1)

• Fold-split calibration: temperature is now fitted on one half of the teacher-labeled points and the conformal quantile on a disjoint half, preserving split-conformal validity (fitting both on the same points made the scores non-exchangeable with fresh records).
• Corrected guarantee wording everywhere: split conformal prediction targets marginal coverage of at least 1-alpha under exchangeability, with abstention on non-singleton prediction sets. Wrong dispatches are <= alpha of ALL records; the conditional-on-dispatch bound (alpha / dispatch rate) is now reported as conditional_error_bound_estimate in the artifact's CALIBRATION metadata.
• Empirical coverage test added: fitted on 300 synthetic points, the measured wrong-dispatch rate on 2,000 held-out points stays within alpha.

Security (Priority 3)

• The lint on generated code now rejects stdlib escape hatches (os, sys, subprocess, socket, urllib, pickle, ctypes, threading, ...) and dynamic-execution calls (eval, exec, open, import, compile, input), with repair-loop feedback.
• Documented plainly: this is validation, not a sandbox - compiling untrusted intent requires external isolation.

Reproducibility (Priority 2)

• New benchmarks/ directory with the exact 398-record routing benchmark, the input-formatting ablation, and methodology notes (the compiler never sees ground-truth labels; the 100% figure is a metric-tuned ceiling; compile variance is documented).

Positioning

STROT - the open decision compiler.

89 tests pass.

v0.8.0 - The decision system, with fitted guarantees

STROT is now a complete decision system: compile intent once with an LLM, decide on every record forever - locally, in type-safe JSON, with a fitted statistical guarantee.

Highlights since 0.2.0

• strot.run() - production JIT: content-addressed artifact cache, schema-drift busting, compile-once/execute-forever (92.5% measured grounded routing, ~60ms/record on CPU, $0 marginal, 0 LLM calls per runtime record)
• strot.decide() - typed questions (choice / score / boolean) over records, typed JSON output with probabilities and confidence
• Golden-record promotion gate - candidate artifacts are executed against your known answers inside the compile repair loop; only passing artifacts are ever served
• Conformal calibration (target_risk=alpha) - temperature scaling + split conformal prediction fitted at compile time from an LLM-teacher-labeled sample; dispatched decisions carry a marginal guarantee (wrong <= alpha), everything else abstains
• Compiler-synthesized routing labels + state projection - measured +18pp accuracy over naive integration on the identical model
• Native Julia-1 integration via its typed-decision API; swappable nlu= slot (local checkpoint or hosted APIs)
• Three synthesizer clients: Anthropic, OpenAI, local claude CLI proxy - resolved lazily, only on cache miss
• Episode logging - every compile recorded to episodes.jsonl
• prefetch_nlu() - bake the decision model into images at build time; fully offline production
• New README and developer guide

83 tests; verified end-to-end in clean Docker containers with the real 144M decision model.

v0.2.0 — Project STROT

First release of the repurposed strot-ai package: an implementation of the STROT framework (arXiv:2505.01636) with compile-time LLM code synthesis and deterministic self-routing NLU runtimes (local CPU transformer or hosted API via Structured Outputs).

Breaking change: this fully replaces the obsolete platform SDK published as strot-ai <= 0.1.3 (strot_ai/strot_cli modules and the strot CLI are removed). Pin strot-ai==0.1.3 if you depend on the old SDK.