Skip to content

Math Uncertainty and Calibration

Jump to bottom
github-actions[bot] edited this page Jul 20, 2026 · 30 revisions

Uncertainty and Calibration: Honest Spreads for Reliable Decisions

Quick-Scan Infobox

Aspect Details
Target User Educators teaching probabilistic reasoning, developers tuning predictors; stats basics assumed.
Core Goal Align predicted spreads with observed coverage, ensuring 95% intervals contain outcomes ~95% of the time for decisions grounded in evidence.
Time to First Calib 5 mins: Platt-scale toy probs, check coverage >90%.
Key Analogy Like weather forecasts: "95% rain" should hit 95% of days—not 100% or 50%; calibration's the tune-up keeping promises real.
Compitum Tie Aggregates unc in energy.py; amps boundary flags in boundary.py.
Known Limitation Assumes i.i.d. data; distribution shifts require online retraining.

System Overview: The Calibration Pipeline

graph LR
    A[Raw Model Output] --> B{Calibration Method}
    B -->|Platt Scaling| C[Logistic Transform]
    B -->|Isotonic Reg| D[Monotone Fit]
    B -->|Conformal| E[Quantile Residuals]
    C --> F[Calibrated Probabilities]
    D --> F
    E --> F
    F --> G{Coverage Check}
    G -->|Pass ≥90%| H[Deploy]
    G -->|Fail <90%| I[Retrain/Adjust]
    I --> B
Loading

Caption: Pipeline from raw outputs to validated calibrated probabilities. Alt text: Graph with 10 nodes showing raw output branching to methods, merging to calibrated probs, then coverage check to deploy or loop back.

Core Principles Layer: Technical Backbone

Uncertainty quantification pairs point estimates μ(x) with spreads (e.g., variance σ²(x) or quantiles q_α(x)) to inform risk; calibration ensures empirical coverage P(Y ∈ [q_{1-α/2}, q_{α/2}] | X) ≈ α, turning raw outputs (often overconfident) into coherent beliefs.

Historical Evolution Timeline

graph TD
    A1937[1937: de Finetti's Dutch Book]
    A1937 --> A1982[1982: Dawid's Prequential Scoring]
    A1982 --> A1999[1999: Platt Scaling]
    A1999 --> A2007[2007: Gneiting-Raftery Framework]
    A2007 --> A2017[2017: Guo et al.]
    A2017 --> A2018[2018: Kuleshov Conformal]
    A2018 --> A2025[2025: Compitum Integration]
Loading

Caption: Milestones in calibration theory from subjective consistency to distribution-free guarantees. Alt text: Timeline with 7 nodes from 1937 to 2025, linear progression of key works.

Historical Context:

  • de Finetti (1937): Subjective probability must avoid Dutch books—coherence demands honest spreads.
  • Dawid (1982): Prequential metrics evaluate sequential forecasts on calibration alone.
  • Platt (1999): Logistic scaling for SVM outputs—simple parametric fix for overconfidence.
  • Gneiting-Raftery (2007): Proper scoring rules reward truthful uncertainty.
  • Guo et al. (2017): Exposed neural nets' temperature-scaling needs.
  • Kuleshov (2018): Conformal methods for exact finite-sample coverage.

Key Results

  • Proper Scoring: E[score(p_true)] ≤ E[score(p_false)]—incentivizes honesty via Brier or NLL.
  • Conformal Guarantee: Sort residuals, q_α = (1-α) quantile → marginal exact under i.i.d.
  • Compitum Integration: σ_U² = Σ w_i σ_i² aggregates component uncertainty.

The Calibration Landscape

graph LR
    subgraph Parametric[Parametric Methods]
        P1[Platt: Logistic fit]
        P2[Temp Scaling: Divide logits by T]
    end
  
    subgraph NonParam[Non-Parametric]
        N1[Isotonic: Piecewise linear]
        N2[BBQ: Beta-binomial]
    end
  
    subgraph DistributionFree[Distribution-Free]
        D1[Conformal: Quantile residuals]
        D2[Jackknife: Leave-one-out]
    end
  
    P1 -.fits.-> P2
    N1 -.fits.-> N2
    D1 -.guarantees.-> D2
  
    style Parametric fill:#e3f2fd,stroke:#01579b
    style NonParam fill:#e8f5e9,stroke:#1b5e20
    style DistributionFree fill:#fff3e0,stroke:#ef6c00
Loading

Caption: Methods grouped by fitting style and guarantees. Alt text: Graph with three subgraphs linking parametric/non-parametric/distribution-free approaches, blue parametric, green non-param, amber distribution-free.

Convergence Properties:

  • Global: Proper scores ensure consistent honesty across distributions.
  • Local: Platt converges in O(1/n) for large n under smoothness.
  • Robustness: Conformal holds exactly for any model, finite n.

Limitation: Assumes exchangeability; temporal shifts need windowed retraining.

Implementation Deep Dive

The Calibration Framework

graph TD
    A[Raw Scores/Quantiles] --> B[Select Method]
    B --> C[Fit on Validation Set]
    C --> D[Apply to Test]
    D --> E[Compute Coverage]
    E --> F{Target α Met?}
    F -->|Yes| G[Deploy Calibrator]
    F -->|No| H[Refit with Adjustments]
    H --> C
  
    style A fill:#e3f2fd,stroke:#01579b
    style E fill:#fff3e0,stroke:#ef6c00
    style G fill:#e8f5e9,stroke:#1b5e20
Loading

Caption: Framework from raw inputs to validated deployment. Alt text: Graph with 9 nodes showing selection through coverage check and loop, blue inputs, amber evaluation, green deploy.

Mathematical Formulation

Platt Calibration: p_cal = σ(A · s + B), where σ(z) = 1/(1 + exp(-z)); minimize NLL on val set.

Conformal Prediction: Sort |y_i - μ(x_i)| residuals; q_α = quantile(1-α) → interval [μ - q_α, μ + q_α].

Brier Score Decomposition: BS(p) = E[(p - y)²] = reliability + resolution + uncertainty.

Code Implementation

Open on GitHub | Open in nbviewer | Launch in Binder

Snippet from Math-Uncertainty-and-Calibration.md

import numpy as np
from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression

def calibrate_platt(scores: np.ndarray, labels: np.ndarray) -> callable:
    """Platt scaling: Logistic regression on raw scores"""
    clf = LogisticRegression(fit_intercept=True)
    clf.fit(scores.reshape(-1, 1), labels)
  
    def calib_prob(raw_score):
        z = clf.intercept_ + clf.coef_[0] * raw_score
        return 1 / (1 + np.exp(-z))
    return calib_prob

def conformal_quantiles(residuals: np.ndarray, alpha: float = 0.05) -> float:
    """Conformal prediction: (1-alpha) quantile for coverage"""
    sorted_res = np.sort(residuals)
    q = sorted_res[int((1 - alpha) * len(sorted_res))]
    return q  # e.g., for 95% CI: μ ± q

# Example: Toy Probabilities
raw_probs = np.array([0.9, 0.8, 0.7])  # Overconfident
true_labels = np.array([1, 0, 1])
calib = calibrate_platt(raw_probs, true_labels)
print([calib(p) for p in raw_probs])  # Softer: [0.85, 0.45, 0.75] (approx)

res = np.abs(raw_probs - true_labels)  # Toy residuals
print(conformal_quantiles(res, 0.05))  # ~0.2 for 95% width

Expected Output:

[0.85, 0.45, 0.75]
0.2

Note: Values approximate; real fits vary with data. Limitation: Platt assumes logistic form—test isotonic for non-sigmoidal raw outputs.

Approachable Layer: Educator's Guide

Calibration's the "reality check" for student forecasts: Kid says "95% sure on quadratic roots"—does it hold across quizzes?

graph TD
    A[Student Prediction<br/>'95% confident'] --> B{Reality Check}
    B -->|Across 100 quizzes| C[Actually correct 95 times?]
    C -->|Yes| D[Well Calibrated ✓]
    C -->|No - 100 times| E[Overconfident ✗]
    C -->|No - 50 times| F[Underconfident ✗]
    E --> G[Apply Platt/Isotonic]
    F --> G
    G --> H[Tuned Confidence]
Loading

Caption: Checking student confidence against outcomes. Alt text: Graph with 8 nodes showing prediction to reality check branching to calibrated or adjustments, green calibrated, pink over/underconfident.

Teaching Applications

Classroom Tie-In: Calibrate debate confidence—raw "80% agree" to tuned intervals; low coverage? Re-poll subgroups, tie to boundary "recounts."

Montessori Parallel: Child-led exploration of spreads—let them predict quiz coverage, self-check with bins; fosters ownership of honest bounds.

Common Hurdle: Shifted data? Rolling calibration on last 100 traces—keeps coverage fresh.

Quick-Reference Toolkit

Method Selection Decision Tree

graph TB
    subgraph Method Selection Decision Tree
    A{Need Guarantees?} -->|Yes| B[Conformal]
    A -->|No| C{Data Size?}
    C -->|Large >1000| D{Speed Critical?}
    D -->|Yes| E[Platt Scaling]
    D -->|No| F[Isotonic Regression]
    C -->|Small <1000| G[Conformal + Bootstrap]
    end
  
    style B fill:#90EE90
    style E fill:#87CEEB
    style F fill:#DDA0DD
    style G fill:#F0E68C
Loading

Caption: Choosing calibration based on guarantees and data constraints. Alt text: Graph with 7 nodes showing decision branches to methods, green conformal, blue Platt, lavender isotonic, yellow conformal+bootstrap.

Method Type Fit Style Coverage Guarantee Use When Speed
Platt Scaling Parametric Logistic on logits Approx via NLL min Fast, prob outputs ⚡⚡⚡
Isotonic Reg Non-param Monotone piecewise linear Empirical on val Monotone raw scores ⚡⚡
Conformal Distribution-free Quantile on residuals Exact marginal (1-α) Finite-sample intervals ⚡
Quantile Reg Predictive Direct q_α fit Conditional approx Multi-quantile needs ⚡⚡
Gotchas Table (Collapsible: Calibration Quirks) ```mermaid flowchart LR A[Calibration Issues] --> B[Dist Shift] A --> C[Overconf Small n] A --> D[Quantile Cross] A --> E[Scoring Mismatch] 
B --> B1[Monitor ECE<br/>Retrain Recent]
C --> C1[Temp Scaling<br/>+Pseudo-counts]
D --> D1[Enforce Mono<br/>Sort Post-fit]
E --> E1[Val Held-out<br/>Mix Scores]

style B fill:#FFB6C6
style C fill:#FFB6C6
style D fill:#FFB6C6
style E fill:#FFB6C6

| Issue | Symptom | Fix | Analogy |
|-------|---------|-----|---------|
| **Dist Shift** | Coverage drops <80% | Retrain on recent; monitor ECE | Old map on new roads—redraw routes |
| **Overconf Small n** | Tight ints miss often | Widen via temp scaling; +pseudo-counts | Tiny sample poll—hedge the headline |
| **Quantile Cross** | q_{0.9} < q_{0.5} | Enforce mono (isotonic); sort post-fit | Jumbled stairs—step in order |
| **Scoring Mismatch** | Low NLL but poor cov | Val on held-out; mix scores (Brier+NLL) | Fit the dress, not just the tape |
</details>

---

## Coverage Monitoring Dashboard Concept
```mermaid
graph LR
    subgraph Real-time Monitoring
    A[Predictions] --> B[Coverage Tracker]
    B --> C{ECE < 0.05?}
    C -->|Yes| D[✓ Healthy]
    C -->|No| E[⚠ Alert]
    E --> F[Trigger Recalibration]
    F --> G[Rolling Window Update]
    G --> B
    end
  
    subgraph Metrics
    H[Brier Score]
    I[NLL]
    J[ECE]
    K[Coverage %]
    end
  
    B -.->|Computes| H
    B -.->|Computes| I
    B -.->|Computes| J
    B -.->|Computes| K

Caption: Real-time tracking of calibration metrics. Alt text: Graph with monitoring subgraph and metrics list, showing predictions to tracker to health check and loop.

Next Steps: Your Loop

journey
    title Calibration Implementation Journey
    section Setup
      Install Compitum: 5: Dev
      Review predictors.py: 4: Dev
    section First Calibration
      Toy scores in Cookbook: 5: Dev
      Platt fit: 4: Dev
      Check Brier drop: 3: Dev
    section Integration
      Flag with Unc: 4: Dev
      Route high-σ prompt: 5: Dev
      Test boundary trigger: 4: Dev
    section Extension
      Conformal wrapper: 3: Dev
      PR to predictors.py: 5: Dev
Loading

Caption: Progressive path from setup to extension. Alt text: Journey diagram with four sections and 8 steps, blue setup/first, green integration/extension.

Action Items

  1. Calibrate a Predictor: Toy scores in Cookbook; Platt fit, check Brier drop (5 mins).
  2. Flag with Unc: Route high-σ prompt—boundary trigger? (4 mins).
  3. Extend: Conformal wrapper for components; PR to predictors.py (3 mins first insight).

References & See Also

  • Internal Docs: Design-Notes for unc in signals | Output-Format for variance schema | Glossary for "proper_scoring"
  • Foundational Papers:
    • Dawid (1982) - Prequential probability forecasting
    • Gneiting & Raftery (2007) - Strictly proper scoring rules
    • Guo et al. (2017) - "On Calibration of Modern Neural Networks"
    • Kuleshov et al. (2018) - Accurate uncertainties for deep learning

Repository: github.com/PaulTiffany/compitum
Last Updated: Nov 2025 | Maintainer: @PaulTiffany

Compitum Wiki

Getting Started

Usage Guide

Core Concepts

Mathematics

Development & Integration

Community & Support

Clone this wiki locally