Publish calibration diagnostics (per-target rows, loss trajectory, skipped targets) with each populace-us release

[PavelMakarchuk]

Problem

populace.calibrate.calibrate() already computes rich diagnostics on CalibrationResult, but none of it is published with a release:

• diagnostics: tuple[TargetDiagnostic, ...] — per-target target, initial_estimate, final_estimate, relative_error, within_tolerance
• loss_trajectory — per-epoch loss (the convergence curve)
• skipped: tuple[SkippedTarget, ...] — targets that failed to compile and why
• options / l0_lambda / n_nonzero — the solver configuration actually used

In packages/populace-data/build/us/build_dataset.py the diagnostics are pushed to internal telemetry (_telemetry.push_target_diagnostics(...)) and then dropped. The published populace_us_2024_calibration.npz keeps only scalars (final_loss, within_10pct, epochs, ...), and the full gate failures lists from GateReport are also collapsed to pass/fail in build_manifest.json.

Why it matters

The calibration-diagnostics dashboard now has a Populace mode (release summary + per-target explorer built on sound_ecps_replacement_comparison.json). What it cannot show without these artifacts:

• Convergence — no loss trajectory means no per-run convergence view (the dashboard has this for us-data runs).
• Skip reasons — "uncompilable targets are skipped and reported, never dropped silently" is a populace-calibrate selling point, but the report never leaves the build machine.
• Calibration-native target fit — the comparison artifact scores populace after a symmetric refit on matched households, which is the right scoring protocol but not the shipped dataset's own calibration diagnostics (initial → final estimate per target, tolerance verdicts).
• Gate failure detail — when a gate fails, the manifest says passed: false without the failing variables.

Proposal

Serialize a calibration_diagnostics.json into releases/<build_id>/ alongside the existing artifacts:

{
  "schema_version": 1,
  "options": {"epochs": 3000, "learning_rate": 0.1, "mass": "free", "max_weight_ratio": 50},
  "loss_trajectory": [0.91, 0.44, "..."],
  "skipped": [{"name": "...", "reason": "missing column ..."}],
  "targets": [{"name": "...", "target": 1.0, "initial_estimate": 0.9, "final_estimate": 0.99, "relative_error": -0.01, "within_tolerance": true}]
}

This is also a DESIGN.md process-rule fit: behavioral evidence about what calibration did should travel with the artifact, not live in telemetry.

Consumer

PolicyEngine/calibration-diagnostics Populace mode — will add convergence and skipped-target views once published.

🤖 Generated with Claude Code