Confidence

Confidence

Confidence in OpenCodifier is multi-dimensional and calibrated — a raw softmax probability is never exposed as confidence (PLANNING.md §73 Rule 8, confirmed by measurement: raw winner probability was uncalibrated at every measured model, ECE 0.048–0.626).

Related pages: Decisions (the policy fields being routed) · Engine (where the gate sits in the pipeline) · Models (calibration artifacts for shipped weights).

• The dimensions
• Outcome routing
• The calibration mandate
• Calibration artifacts and their gates
• Selective prediction (0.6.x)
• Per-rung honesty

The dimensions

A ConfidenceReport is not one number. The report carries:

Dimension	What it measures
Top probability	calibrated confidence of the winning answer
Margin	top1 − top2 gap — a near-tie is a different animal from a landslide at the same top probability
Entropy	normalized answer entropy (entropy / log2(k)) — distributional shape
OOD	out-of-distribution evidence; policy-selectable mode (D28): answer_entropy (default) or lexical_band (1 − lexical_coverage of the best-scoring document; no lexical grounding ⇒ maximal OOD)
Verifier agreement	whether an independent second pass agrees (confidence-gated — never run on every request)

The gate cascade is demote-only: a gate can demote an outcome (accept → verify/abstain), never promote one.

Outcome routing

ConfidenceReport::outcome_for(policy) routes every answer (frozen v1 semantics):

calibrated confidence < abstain_below            → Abstain
calibrated confidence ≥ min_confidence
  AND risk ∈ {Low, Medium}                       → Accept
everything else                                  → Verify

• High/Critical risk never auto-accepts — those risk levels always route to at least Verify.
• Abstain and Escalate are successful outcomes, never errors: HTTP returns 200 with the typed outcome; the CLI's exit-code contract maps them to exit 2 (or 0 with --abstain-is-success).
• An elected abstain_candidate is a terminal abstention regardless of confidence — a 0.99-confidence "none of these" is never re-derived into an acceptance.
• A boolean threshold flip keeps the model's measured distribution exactly as it is; the answer describes the verdict actually given, and the gate reads the honest confidence of the minority mass.

The calibration mandate

Raw softmax probability ≠ calibrated confidence. Calibration is mandatory before any confidence leaves the runtime.

The engine's Calibration seam (D15):

Scheme	Behavior	Cache identity
IdentityCalibration	raw, version 0 — reported as calibrated_confidence = raw, calibration_version = "none"	folds into cache keys so calibrated and uncalibrated results can never be conflated
TemperatureCalibration	per-question-kind temperatures from a validated versioned JSON artifact (format_version 1, scheme "temperature", deny_unknown_fields, calibration_version ≥ 1)	artifact version folds into every cache key — adopting a refit invalidates cached decisions
ProofAwareCalibration	single-entry distributions (exact proofs, p = 1.0) pass through raw; multi-entry distributions delegate to the inner calibration	the wrapping ladder's id re-keys

Fitting is offline (the fitter lives in the benchmark tooling); the artifact — not the fit — is what ships.

Calibration artifacts and their gates

An artifact earns deployment only if it survives both gates:

1. ECE gate (D15 amendment, 2026-09-28): the fit must not worsen expected calibration error. In-sample NLL improves on nearly any one-parameter fit, so ECE is the shape check. This gate retired the engine arm's global temperature fit (it worsened ECE on the bimodal proof/delegate distribution) and removed the artifact that described a superseded stack.
2. Risk–coverage gate (D15 amendment, 2026-10-07): calibration error and selective prediction measurably diverge, so the fit provenance also carries aurc_before / aurc_after / accuracy_at_coverage, and both the fitter's ship gate and the engine's load gate refuse an artifact whose AURC worsens beyond rounding slack (1e-9). A calibration artifact now earns deployment by its risk–coverage curve, not a single operating point.

For temperature scaling the risk–coverage leg is currently a witness (a monotone rescaling cannot reorder items); it becomes load-bearing with the first non-monotone scheme. The numbers of record live in benchmarks/decision-model/results/CALIBRATION.md.

Selective prediction (0.6.x)

v0.6.1 added selective-prediction end to end (see the changelog for the commit receipts):

• a label-overlap preflight that abstains before extraction guesses,
• a selective-prediction evidence floor,
• boolean gate fit with elicited abstain (the abstain_candidate mechanism above),
• risk-coverage acceptance for calibration artifacts.

Read together: the runtime can now learn when not to answer as a first-class, measured capability rather than a threshold setting.

Per-rung honesty

Different rungs produce different kinds of evidence, and the runtime refuses to pretend otherwise:

• Relational proofs are p = 1.0 by construction — exact proofs over extracted facts are the only source of proof-grade confidence in the zero-ML stack (D35).
• The embedding rung's probability is uncalibratable — no finite temperature helps (degenerate fit), so no artifact is shipped for it; its scores are ordering evidence only, and the measured gate is a margin floor (top1−top2 ≈ 0.018), not a probability gate.
• The lexical rung is structurally bounded below 1.0 (D35) so lexical overlap cannot impersonate a proof.
• The model rung carries a fitted per-kind temperature; server-reported probabilities are never exposed raw (D26).

This is why the escalation ladder (Engine) resolves policy and calibration per rung: a single request-level policy cannot gate rungs whose confidences are not commensurable.