Skip to content

Add fiscal diagnostics comparison tool#190

Merged
MaxGhenis merged 1 commit into
mainfrom
codex/target-audit-tool-20260626
Jun 26, 2026
Merged

Add fiscal diagnostics comparison tool#190
MaxGhenis merged 1 commit into
mainfrom
codex/target-audit-tool-20260626

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a diagnostics-only tool for comparing two Populace US calibration_diagnostics.json files on a strictly shared target surface
  • compute the same sqrt-value weighted, 50/50 amount/count, capped-MAPE objective used by the US fiscal refresh
  • emit JSON, CSV, and Markdown artifacts with family contributions plus top candidate regressions/improvements
  • reject duplicate diagnostic target keys and reject shared target names whose target values differ, so stale same-name comparisons cannot be treated as release evidence

Validation

  • uv run ruff format tools/compare_us_fiscal_diagnostics.py packages/populace-build/tests/test_compare_us_fiscal_diagnostics.py
  • uv run ruff check tools/compare_us_fiscal_diagnostics.py packages/populace-build/tests/test_compare_us_fiscal_diagnostics.py
  • uv run python -m pytest packages/populace-build/tests/test_compare_us_fiscal_diagnostics.py

Real comparison guardrail

A previous candidate/incumbent smoke used diagnostics with overlapping target names but changed capital-gains target values. The tool now correctly rejects that comparison with ValueError: Cannot compare diagnostics with changed shared target values. The incumbent must be rescored on the exact candidate/current target surface before using this tool for release certification.

@MaxGhenis MaxGhenis force-pushed the codex/target-audit-tool-20260626 branch from 1c9bda3 to d20a117 Compare June 26, 2026 19:13
@MaxGhenis MaxGhenis merged commit dbaa97c into main Jun 26, 2026
4 checks passed
@MaxGhenis MaxGhenis deleted the codex/target-audit-tool-20260626 branch June 26, 2026 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant