Skip to content

v0.2.0

Choose a tag to compare

@aliasfoxkde aliasfoxkde released this 29 Sep 06:21
· 498 commits to main since this release

Decision-runtime feature release: the decision-model benchmark and D16 four-tier board, confidence truthfulness (calibration seam, uncertainty gates, live OOD score), the relational solver as the engine's default zero-ML stack, and the rmcp 2.2 stdio MCP surface.

[0.2.0] — 2026-09-28

Four plan phases in one minor bump: the decision-model benchmark and
D16 board (Phase 13), confidence truthfulness (Phase 14), the
relational solver that became the engine's default zero-ML stack
(Phase 15), and the MCP surface (Phase 10).

Added

  • Decision-model benchmark (PLAN Phase 13) — built and swept to 48
    runs: benchmarks/decision-model/ holds a byte-locked 120-item
    candidate-conditioned suite (metadata_match / lexical_semantic /
    relational_compositional) and measured arm types (the engine's
    lexical pipeline, embedding zero-shot over three backends, Laya-421M,
    llama.cpp parallel-decision constrained scoring across the GGUF
    board, and chat-only screens for forks without a decision arm), with
    the committed summary + model manifest. Measured and recorded: the
    2-bit cliff (0.617 at 4B), the 9B IQ-quant latency reversal, the
    ≤350M tiny-decoder graveyard, and Jev-Style-0.8B-Decision-v3 as an
    interface-mismatch row (0.217 — verdict-slot readout, zero transfer
    to candidate-id path scoring; labeled in the summary). Extension
    arms (2026-09-28): Bonsai-4B ternary is the board's best
    accuracy-per-byte (0.650 from 546 MiB) at 4B-class latency;
    Ternary-Bonsai-8B is unmeasurable on the CPU host (>10 CPU-hours per
    decision request / unreadable tensors); LFM2.5-2.6B (0.608) is
    dominated by Qwen3.5-2B; the DavidAU X12 NEO MAX merge (0.667 @
    8.35 s) is latency-toxic. Ternary second pass (2026-09-28): Bonsai-8B
    Q1_0 hits a measured prefill cliff (≈1.5 s/token past a ~30-token
    knee — suite projects to 9–13 h) and Ternary-Bonsai-2-27B PTQ1_0 is
    unloadable (ggml type 143, prism-ml-fork-only), so on-CPU ternary
    viability in the ad129b0 build is 4B-and-below. Embedding-rung
    bake-off (2026-09-28): gte ONNX-fp32 ties torch to the digit at 4.2×
    the CPU speed (811 ms vs 3375 ms per item) and becomes the rung's
    runtime; int8 dynamic quantization rejected (0.442 for −17%
    latency); EmbeddingGemma-300M Q8_0 via llama.cpp recorded (0.500 /
    ECE 0.256); fp16 export unloadable (REPORT F20); blockwise int4 (the
    Q4_0 analog via ORT MatMulNBits, block 32) measured quality-free AND
    speed-free — 0.575 / ECE 0.330 identical to fp32 at 226 MB with no
    latency gain, so fp32 stays the rung's runtime and the q4 build is
    the RAM-bound fallback (REPORT F21; q4f16 untestable on CPU). Raw
    winner probability measured uncalibrated everywhere (ECE
    0.048–0.626), so D15 calibration still blocks any confidence
    exposure, and the relational class stayed escalation territory for
    every model arm until the relational solver (below). Runner
    hardening: per-request --timeout (9B tails exceeded the old fixed
    600 s and killed arms mid-run), chat-only screens included in the
    summary, and per-run caveat annotations. Full narrative record
    committed as results/REPORT.md + results/charts/ (four
    deterministic SVG charts rendered from the run JSONs by
    runner/plot.py) — methodology, complete board, quant ladders,
    findings F1–F17, threats to validity, and reproduction commands.
  • Confidence truthfulness (PLAN Phase 14): the §73 calibration mandate
    and the dead OOD channel are both closed. The engine gains a
    Calibration seam — identity (raw, version 0) by default,
    TemperatureCalibration from validated versioned artifacts whose
    calibration_version folds into cache keys — plus §19 uncertainty
    gates on DecisionPolicy (entropy ceiling / minimum margin / OOD
    ceiling, disabled by default, byte-stable canonical form) fed by a
    live distributional OOD score (entropy / log2(k)) traced alongside
    the calibrated confidence. Fitted temperature artifacts for the six
    D16 tier / rung arms ship under
    benchmarks/decision-model/results/calibration/ (offline fitter
    runner/fit_calibration.py; evidence in results/CALIBRATION.md):
    LLM tiers are overconfident (T 0.84–1.67, frontier MiMo-9B ECE 0.048
    → 0.026), and the embedding rung's fit is degenerate — no finite
    temperature calibrates gte zero-shot scores, which are ordering
    evidence only, so no artifact is shipped for it.
  • Relational solver (PLAN Phase 15): state text is mined for
    relational facts with an exact grammar ("X depends on Y", "X is
    healthy", "X: healthy, degraded, down", "X comes back online only
    after Y"); three general operators (root cause by transitive
    dependency closure, healthiest group by strict count, first restored
    by ordering) prove an answer over the whole extracted structure —
    and answer only when the proof is unique, inside the candidate set,
    and agreed by every operator, delegating to the lexical classifier
    otherwise. Measured on the byte-locked benchmark suite:
    relational_compositional 0.350 → 0.950, blended 0.483 → 0.683, p50
    5.3 ms → 1.3 ms, other classes bit-identical, determinism replay
    exact — the 0.50 relational ceiling that held for every model arm is
    broken at zero cost by proof rather than likelihood (REPORT F22).
    Calibration of record for the engine arm stays identity — the global
    temperature fit worsens ECE on the bimodal proof/delegate
    distribution, and the fitter now refuses to ship any artifact whose
    fit does not improve ECE (the retired builtin-lexical-v1 artifact
    was removed with the stack it described).
  • MCP surface (PLAN Phase 10): opencodifier-mcp — six decision tools
    (codify_decide, codify_batch, codify_graph, codify_validate,
    codify_verify, codify_explain) served over stdio with rmcp 2.2.0
    (D1's pin, honored; the 3.x major line is not adopted — D17), wired
    as opencodifier mcp serve. Same engine as HTTP (EngineHandle),
    same error envelope and stable codes (one MCP-specific code:
    mcp.batch_too_large), same deterministic traces with no
    chain-of-thought; batch items are decided independently under a
    16-request cap.

Changed

  • The engine's default zero-ML stack is the relational solver over the
    lexical classifier (Phase 15); D16 amended ×4 into a four-tier
    scheme: MiMo-V2.6-9B Q3_K_S is the frontier (0.817, ECE 0.048 best,
    first arm over the old 0.50 relational ceiling at 0.525 — MoE,
    14.3 s p50, verifier tier); Qwen3.5-4B is the interactive reference
    (0.800 @ 4.9–6.8 s, superseding Qwen3.8-4B-Distill); Qwen3.5-2B
    stays balanced (0.725, best ECE 0.062) and Qwen3.5-0.8B fast
    (0.650 @ 613 ms — now undercut by the engine floor).
  • Cache-key honesty (D6): the engine now derives the identity's model
    id from the live classifier, so wrapper ids like
    relational-v1|builtin-lexical-v1 invalidate cached decisions on
    swap without caller bookkeeping. Two shared-surface folds landed
    with the MCP work: the engine's graph-document shim (GraphDocument)
    is now public and the HTTP surface's private mirror was deleted, and
    the execution-report JSON projection moved into the engine
    (opencodifier_engine::report) so CLI --trace and MCP
    codify_explain cannot drift.
  • Aegis baseline regenerated at the release commit (2,370 entries;
    line-shift re-flags after the CI-clippy allow fix in c09e39d turned
    24 already-triaged findings into false "new" ones on pipeline
    aaa2eefa).