fix(polities): make an aggregation bucket sum, under exactly one area label - #591
Conversation
`.aggregate_to_polities()` grouped by the member's `polity_name` as well as `polity_area_code`, so a bucket folding members that resolve to different polities was never summed: it came back as several rows under one `area_code` carrying different `area` labels. That is live on the shipped crosswalk -- bucket 206 folds FAOSTAT 276 Sudan and 277 South Sudan, which resolve to two polities from 2012 -- and it is the reason #480 had to be reverted (#561). The label is now derived after the sum, from the bucket's own code, which is what `polity_bucket_coverage()` reports and what the reporting columns resolve. `.read_crop_residues()` duplicated the aggregation inline and now shares the same two helpers, so the two sites cannot emit two `area` vocabularies for one `area_code` -- the shape that dropped 702,166 rows in whep#382. Measured over the real pins, the four split sources lose exactly their duplicate-key count in rows and keep their totals to the digit. Closes #563 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Verified independently on I filed #563 as "the prerequisite for redoing #480". That framing was wrong, and your reproduction is why. Bucket 206 folds FAOSTAT 206/276/277, those resolve to three polities from 2012, so the split has been live on shipped data the whole time. Reproduced on
|
main |
this PR | ||
|---|---|---|---|
| goats 2018 (item 976) | 14,449,249 | 55,295,249 | South Sudan alone → Sudan + South Sudan |
| sugar cane 2019 (item 156), tonnes | 10,898,000 | 5,449,000 | exactly 2× → the reported value |
| sugar cane 2019, ha | 142,200 | 71,100 | the area was doubled too |
Both of your headline numbers land exactly. The other bucket-206 livestock items move the same way — item 1053 goes 23.4 M → 62.9 M, 1016 goes 17.0 M → 48.8 M — so Sudan's herd was being dropped across the board.
Data was being silently dropped in one direction and doubled in the other, in the same bucket, depending on which member reported.
On my guard being too narrow
The invariant test I added in #566 passes on main — and it should not have. It uses areas 212 and 999, which resolve to the same polity there, so it only ever caught the #480 shape and never the live bucket-206 case. Your reader-level measurement (42 split area-years, 7,939 duplicate keys across eight readers, → 0) is the check that actually covers it. Worth folding that breadth into the guard so it cannot narrow again.
I also confirmed the published frame carries 0 duplicate keys on main (6,170,595 rows), which is consistent with your account rather than contradicting it: the split never survives to the output, it is resolved by dropping or doubling upstream, which is exactly why no downstream check saw it.
Full suite 5781 pass, 0 fail.
What I am not deciding
Your three open questions stay open and I have not pre-empted any of them: bucket 206's label (SUD-1956-2011 vs area 276's polity) is #414's call; whether to re-land the un-fold is #459's, now unblocked because the grouping no longer splits a bucket; and options(whep.unfold_rest_of_world = TRUE) stays off.
Also filing your #590 finding as worth acting on — two tests mocking .polity_bridge() believing it controls .aggregate_to_polities(), when it resolves through .polity_crosswalk() instead, is a mock that has silently never done anything.
What was wrong
.aggregate_to_polities()grouped by the member'spolity_namealongsidepolity_area_code, then renamed the pair toarea/area_code. So a bucketthat folds members resolving to different polities was never actually
summed: it came back as several rows under one
area_code, under differentarealabels. Mass conserved,polity_area_codewas untouched, bucketmembership was identical — which is why #480 passed every check anyone ran and
still had to be reverted (#561).
The issue frames this as a blocker for a future un-fold. It is not: it is
live on the shipped crosswalk today. Bucket 206 "Sudan (former)" folds
FAOSTAT areas 206, 276 Sudan and 277 South Sudan, and from 2012 those resolve
to three different polities. So
mainhas been emitting a split bucket forthirteen years of four different sources, and the split was not cosmetic —
downstream it both dropped and doubled data (numbers below).
.read_crop_residues()(R/build_cbs.R) duplicated the same aggregationinline, which is the second half of the hazard: two sites emitting two
areavocabularies for one
area_codeis exactly what dropped 702,166 rows in #382.Evidence it reproduced BEFORE the change
Driving the real readers over the real pins, at the base commit, full range:
(area_code, year)under >1 label(area_code, year, item, element, unit)keysfaostat-fbs-newfaostat-productionfaostat-trade-totalsfaostat-emissions-livestockfaostat-fbs-old,-cbs-old-crops,-cbs-old-animal,-production-oldEvery split is bucket 206, labels
South Sudan | Sudan. Re-measured afterrebasing onto today's
main(which had moved by #408/#405): identical.The consequence, traced to the raw pin rather than asserted:
faostat-productionbuild_primary_production()beforeWhat I changed
polity_nameis no longer a grouping key. The label is attached afterthe sum, from the bucket's own code resolved for that year — which is
what
polity_bucket_coverage()already documents a fold as being called andwhat
add_reporting_polity_columns()already resolves, so the aggregator nowagrees with both instead of using a third rule. That is option 2 of the
issue's three, carrying option 1's labelling rule.
.bucket_area_labels()and.apply_bucket_area_labels(), inR/polity_folds.R;.read_crop_residues()uses them too, so the two aggregation sites cannot diverge.
or reorder a row — a missing label costs a label, never a row. That is the
polities: link the package to whep-polities and stop resolving to dead polities #382 failure mode made structurally impossible rather than tested for.
whose period has ended) falls back to the member label, deterministically by
lowest
area_code, so no row is left unlabelled.How I verified
Structure, not just totals — #480 is the reason:
keys 7,939 → 0, and each source's total is unchanged to the digit
(e.g.
faostat-fbs-new948,260,434,000 before and after). Row counts fall byexactly the duplicate-key count: fbs-new −2,056, production −2,000,
trade-totals −3,739, emissions-livestock −144.
build_primary_production(), full range: 6,170,595 rows before and after,key set identical (0 keys added, 0 removed), 1,673 keys change value, all
in 2012-2023 and all in bucket 206 except two derived per-head ratios for
Belgium in 2023 (0.0062 and 0.0006). Both directions of change verified
against the raw pin (table above).
build_commodity_balances()on real data, 2005-2020 (a full-range buildwas OOM-killed on a machine shared with nine agents; this range covers every
affected year): 1,276,337 → 1,276,340 rows, 678 keys change of which 559 are
bucket 206. Every other area moves 43.4 t in total across 119 keys, largest
single move 3.91 t — 4e-9 % of the build, trade-balancing knock-on.
Element totals:
stock_variation+1.79 %, everything else within 0.03 %.reporting_polity_codefor bucket 206 isSUD-1956-2011before and after.area_code-one-areaguard intest_read_raw_inputs.R(the oneadded after the fix(polities): give regions.csv's grid countries and pre-2011 Sudan their real polities #480 revert) still passes, and two more assertions now pin the
live 206 case and the labelling rule.
Gates:
air format .clean;devtools::document()committed;lintrwith theCI config reports 0 lints; every
man/*.Rdis in_pkgdown.yml;devtools::test()5,781 pass / 0 fail / 8 skip (run on the rebased base).Assertions that fail without the R change (verified by stashing
R/andre-running): 13 — 5 in
test_read_raw_inputs.R, 8 intest_polity_folds.R.Moves published values
Yes, and only for bucket 206. The move is a correction in both directions and
each direction is checked against the raw FAOSTAT pin, not against a prior WHEP
build.
NEWS.mdcarries the magnitudes. Nothing else in the pipeline changes bymore than 3.91 t.
One existing test changed its expectation:
.aggregate_to_polities warns when it folds Sudan into bucket 206assertedarea_code == c(206, 206)— two rows for one bucket. It now asserts one row of3,405,356, because that is what the warning next to it has always claimed was
happening.
What I deliberately did not do
for and nothing more: with the grouping fixed, a polity-level un-fold no longer
splits a bucket. Which areas should be un-folded is still Fix polity_area_crosswalk for regions.csv's 9 present-day-invalid area codes #459's call. Verified
on the fixture crosswalk that already ships in
test_polity_folds.R: bucket900 folding members 901 "X" and 902 "Y" onto their own polities now comes out
as one row labelled "Aggregate" — the exact shape fix(polities): give regions.csv's grid countries and pre-2011 Sudan their real polities #480 shipped.
Sudan under a single polity, which is Bucket 206 sums Sudan + South Sudan but is attributed to SDN-2011-2025: the value and its polity describe different territories #414 and a science decision. This PR only
makes the sum actually happen and gives it one name.
options(whep.unfold_rest_of_world = TRUE)(option 3 in the issue) isunchanged and still off by default; it now labels each promoted bucket from its
own crosswalk row, as before.
.fill_with_proxies()joinspopulation and land by
c("year", "area"), and.read_land_areas()labelswith the FAOSTAT
area_namewhile the aggregator labels with the polity name:63 of 166 CBS-side labels in 1961 have no counterpart in
land_wide, includingChina (PRC), Czechoslovakia and Belgium-Luxembourg. That is pre-existing,
unchanged by this PR (only bucket 206's label moved, and only for 2012+), and
already tracked in Key .fill_with_proxies() on polity_area_code instead of area name; decide what an aggregate's proxy means #493 and Two 'area' vocabularies: .resolve_hist_trade_polities() labels with area_name, everything else with polity_name, and 68 buckets disagree #584.
The open decision
Bucket 206 is labelled
SUD-1956-2011"Sudan (1956-2011)" — a polity that hadended by the years in question — because that is the polity its own code
resolves to, and because the pre-2012 half of the same series was already
labelled that way. The alternative is to keep labelling the post-2011 rows
"Sudan" (area 276's polity), which is what the split accidentally did for
roughly two thirds of them. I took the bucket's own label because it is the
rule the rest of the package already applies, and because a join key that flips
mid-series is the defect this PR exists to remove — but the choice of which
polity a two-territory bucket should name is #414's, not mine.
Closes #563
Part of the polity migration epic #458.
🤖 Generated with Claude Code