Calibrate LA council tax (band counts + net £) and fix national gross/net

[vahid-ahmadi]

What this PR does

Calibrates both FRS council tax data points at LA level, addressing the 28 Apr standup ALIGNED decision ("the model will calibrate the two FRS data points as the council tax information is provided after deductions").

Three new column families added to datasets/local_areas/local_authorities/loss.py:

FRS data point	Matrix column	Target column	y formula
Council tax band	1[council_tax_band == B] for B in A–H	voa/council_tax/{A..H} (8 cols)	VOA chargeable dwellings per band per LA
Council tax £ paid (net of CTR)	council_tax_less_benefit (= gross − CTR benefit)	housing/council_tax_net	MHCLG taxbase-after-CTR × Band D (E); WG Council Tax Income (W)

Plus a national-level fix: targets/compute/council_tax.py now uses council_tax_less_benefit instead of council_tax for the OBR obr/council_tax target (which is "Total net council tax receipts"). Both sides of that constraint are now net of CTR.

LAs missing either input have their y cell left as NaN; the calibrator (utils/calibrate.py) masks NaN cells out of the loss so missing-source LAs don't contribute to training on those targets. No fabricated national-share fallbacks; no hard-zero NI targets — only directly observed cells train the calibrator (commit 96f5707).

Lineage caveats (flagged in review by @MaxGhenis)

Both council-tax LA targets are derived/proxy targets — products / rescalings of observed inputs, not directly observed LA totals. The PR earlier described them as direct; that was overstated.

Band counts (voa/council_tax/{A..H}):

• Target: VOA dwelling stock per band per LA (E&W only).
• Matrix: policyengine-uk household council_tax_band.
• Drift: dwellings ≠ households; VOA stock includes exempt / empty / second-home dwellings without a household; banding ratios differ in Scotland (post-2017) and Wales (Band I, England has none).

Net £ paid (housing/council_tax_net):

• Target: MHCLG taxbase × Band D (E), WG-published net council tax income (W). Taxbase is Band D equivalent dwellings adjusted for ~7 discount/premium/exemption classes.
• Matrix: FRS-reported council_tax_less_benefit (household-reported gross less reported CTB).
• Drift: same intent (what households pay net of CTR) but different construction paths and underlying microdata.

A separate policy question — whether derived/proxy targets like these should sit at full training weight alongside directly observed targets (HMRC SPI, ONS pop, DWP UC, VOA dwellings) — is being tracked in #381 and is not blocking this PR.

Closes #370.

Sources

Country	Source	Coverage
England (296)	MHCLG Council Taxbase 2025, Table 1.35 "Tax base after allowance for council tax support" × LA Band D amount	full
Wales (22)	Welsh Government Council Tax Levels April 2026 to March 2027, Table 3 "Council tax income (£m)" — already net of CTR	full
Scotland (32)	None wired; cells masked (NaN, excluded from loss). Follow-up: parse Scottish Government Council Tax Datasets.	masked
NI (10)	No council tax (domestic rates instead) — cells masked	masked

Sanity check. Across the 318 LAs with a directly observed net £ figure, the per-LA targets sum to ~£49.9bn (England £47.4bn + Wales £2.45bn). This roughly reconciles with MHCLG's published England-only Council Tax Requirement of £45.86bn (small gap from year mismatch — 2025 taxbase × 2026-27 Band D). The reconciliation is for sanity only; the calibration constraint operates per-LA, not on the national aggregate.

Files

New / modified

• policyengine_uk_data/storage/la_council_tax.csv — adds total_council_tax_net column for E/W LAs.
• policyengine_uk_data/targets/sources/la_council_tax.py — load_la_net_council_tax() helper + Target objects named housing/council_tax_net/{code}. Module docstring documents derived/proxy nature + lineage caveats.
• policyengine_uk_data/targets/compute/council_tax.py — switches OBR national matrix col from council_tax (gross) to council_tax_less_benefit (net) so both sides of the constraint are net of CTR.
• policyengine_uk_data/datasets/local_areas/local_authorities/loss.py — housing/council_tax_net block immediately after the band-count block. Both blocks leave missing-source cells as NaN; the calibrator masks them.
• policyengine_uk_data/utils/calibrate.py — calibrator updated to mask NaN cells out of the per-LA loss so sparse local targets are first-class.
• policyengine_uk_data/tests/test_la_loss_council_tax.py — layer-1 CSV/coverage tests + layer-2 FRS-fixture-gated wiring + calibratability + NaN-masking assertions.
• policyengine_uk_data/tests/test_calibrate_save.py — toy calibrator regression locking in the NaN-masking contract.
• policyengine_uk_data/tests/test_obr_council_tax.py — 3 tests pinning the net-variable contract for the national OBR target.

Tests

• CSV-level: total_council_tax_net column present, England + Wales fully covered, Scotland + NI absent (masked territory), value range £2m–£1.5bn (lower bound for Isles of Scilly), covered total in £43–50bn ballpark.
• Loss-matrix wiring (FRS-fixture-gated): housing/council_tax_net column present in matrix and y, matrix col equals sim.calculate("council_tax_less_benefit"), direct cells finite, English LA y matches CSV exactly, Scotland + NI cells are NaN (masked, not fabricated), covered-LA target sum within 0.3–3× of weighted initial covered-LA net CT (calibratability sanity).
• Calibrator masking (toy): NaN local target cells stay finite in the produced weights — they don't propagate NaN through the loss.
• National OBR compute (light): compute_obr_council_tax returns council_tax_less_benefit not council_tax; country masks apply correctly; gross variable not queried.

Full run incl. adjacent suites: no regressions.

Out of scope (follow-ups)

• Scottish per-LA net council tax sourcing (Scottish Government Council Tax Datasets).
• Scottish Assessors per-LA chargeable-dwellings to fill the band-count gap.
• LA calibration: unify missing-source handling across loss.py blocks #381 — LA calibration: unify missing-source handling across loss.py blocks (NaN-masking vs national-share fallback).

Related

• Add main residence value to LA calibration #371 — LA-level main residence value targets (companion PR; demoted to validation-only by Max in c330b44).
• LA calibration: unify missing-source handling across loss.py blocks #381 — tracks the broader question of whether NaN-masking should apply to the existing tenure / rent / ONS-income blocks too.
• 28 Apr standup decisions: "Calibration approach for FRS data" + Max's "calibrate on observable LA-level housing indicators" framing.

[vahid-ahmadi]

Edits:

1. Both FRS data points now calibrated at LA level

FRS data point	Where
Council tax band	voa/council_tax/{A..H} — 8 cols (already in this PR)
Council tax £ paid	housing/council_tax_net — new in this commit

2. Net of CTR alignment on both sides

• Matrix col: council_tax_less_benefit (= gross − CTR benefit), not council_tax.
• Target value: directly published net council tax requirement per LA.

3. Net receipts data investigated and used directly (no derivation)

• England (296 LAs): MHCLG Council Taxbase 2025, Table 1.35 Tax base after allowance for council tax support × Band D. Sums to £47.4bn vs MHCLG's published Council Tax Requirement of £45.86bn (within 3.4%; year-mismatch).
• Wales (22 LAs): Welsh Government Table 3 Council tax income (£m) — already net.
• Scotland (32) + NI (10): fallback to national × la_household_share, same pattern as the band-count and tenure targets. Scottish source is a separate follow-up.

4. Same shape as the existing rent / tenure / housing/main_residence_value (#371) blocks — directly observed LA quantity, no national apportionment, na-fallback for missing LAs.

One thing I flagged but did not fix here: at the national level, the OBR obr/council_tax target value is "Total net council tax receipts" but the matrix col in targets/compute/council_tax.py:24 is council_tax (variable labelled "Gross amount, before discounts"). Either the variable label is wrong (FRS encodes net, label is misleading) or there's a real gross/net mismatch in the national reweighter. Worth verifying separately — it maps to the "[Vahid Ahmadi] Review Data" action item.

[vahid-ahmadi]

Folded the national fix into this PR (commit 3277a21):

targets/compute/council_tax.py:24 was passing the gross variable (council_tax) as the matrix col for the net OBR target (Total net council tax receipts). Now uses council_tax_less_benefit so both sides are net.

Order-of-magnitude check:

• Σ w × council_tax (gross): ~£55bn
• Σ w × council_tax_less_benefit (net): ~£47bn
• OBR Table 4.1 net target: ~£44bn

Three regression tests added (test_obr_council_tax.py) pinning the net-variable contract — a future refactor that reintroduces the gross variable will fail at test time, not silently in production.

Effect: closes the "[Vahid Ahmadi] Review Data" loop. National council tax calibration is now consistent with the FRS-net framing, and pairs naturally with the LA-level housing/council_tax_net target this PR adds (both use council_tax_less_benefit).

[MaxGhenis]

Follow-up on the direct-target discussion: I pushed 96f5707 to stop fabricating local council-tax targets where no direct source cell exists. Missing CT band/net cells now stay NaN and calibrate_local_areas masks NaN local target cells out of the loss, rather than filling Scotland/NI with national-share values or hard zeroes. This preserves direct source cells and avoids treating NI zeroes as training constraints when the matrix side may not be zero.\n\nThis also adds a toy calibrator regression for sparse local targets. Remaining source-cleanup direction: prefer direct council-tax requirement/income where available, and add Scotland band/net sources before training on those LA cells.\n\nVerification: uv run pytest policyengine_uk_data/tests/test_calibrate_save.py policyengine_uk_data/tests/test_la_council_tax_targets.py policyengine_uk_data/tests/test_la_loss_council_tax.py policyengine_uk_data/tests/test_obr_council_tax.py -q; ruff check/format on touched files.

[vahid-ahmadi]

@MaxGhenis — nit on 96f5707.

is_ct_la may now be dead defensive code. With the switch to NaN-masking, NI LAs naturally have no total_council_tax_net and no count_band_* values in the CSV, so has_count and has_ct_net are already False for them — the cells would be NaN without the explicit is_ct_la gate. Worth checking whether the gate is still doing useful work or can be removed.

Also is_ct_la = ct_merged["has_council_tax"].fillna(True).astype(bool).values defaults missing flags to True, which seems backwards directionally — a missing flag should probably mean "unknown / err on the side of masking", not "yes train on this".

Not blocking.

[MaxGhenis]

Addressed in 7f0c3e2: removed the redundant is_ct_la gate from the LA council-tax loss matrix.

After the NaN-masking change, direct source-cell availability is enough: band targets train only where count_band_{A..H} is non-null, and net council-tax targets train only where total_council_tax_net is non-null. NI already has NaNs for those cells, so it remains masked without a separate has_council_tax flag or a missing-flag default.

Verification:

• uv run pytest policyengine_uk_data/tests/test_la_loss_council_tax.py policyengine_uk_data/tests/test_calibrate_save.py -q
• uv run ruff check policyengine_uk_data/datasets/local_areas/local_authorities/loss.py
• uv run ruff format --check policyengine_uk_data/datasets/local_areas/local_authorities/loss.py