fix(#323): add a condition index level to Uganda crop_production

[ligon]

The defect

The UNPS post-harvest module asks question 6 as one compound question — how much did you harvest, in what unit, and in what condition/state. A household that takes some of a crop off a plot fresh and the rest after drying it answers twice.

crop_production was keyed (t, i, plot, j, u, season) with no level for the condition, so those answers landed on the same index tuple and the de-duplication block at the end of uganda.crop_production_for_wave summed them. Dry weight was added to fresh weight.

2011-12  HH 1033000506  plot -1-6  Coffee  Kilogram (KG)
   240 kg  "Dry after additional drying - in pods or shell/husks"  month 6
   100 kg  "Fresh/raw harvested - in pods or shell/husks"          month 6
   -> summed to a single 340 kg row

340 is not a quantity of anything. 3 654 index groups were affected. The quietest instances were 2009-10 and 2010-11: those waves have no harvest-month column, so every one of their collisions looked like an exact duplicate of an identical record.

The fix

Grain is now (t, i, plot, j, u, condition, season).

Both vintages use the same 20 integer codes.

vintage	shape	condition source
2009-16	LONG — one column group, multiple source rows per plot-crop	a value in a5aq6b / a5bq6b
2018-20	WIDE — parallel _1/_2 slots	a value in each slot's own 6c column

2019-20's _1/_2 are not an ordinal "first/second harvest": the Stata variable labels read literally 6c. Condition/state of crop harvested (2018, full harvest, condition1) / (… condition2). So one harvest_conditions table in categorical_mapping.org serves every wave — no harmonisation between vintages was needed or invented, only cosmetic label drift (en-dash vs hyphen, mojibake in 2015-16 AGSEC5B, code 99 reading "Others" in 2011-12 and "Not Applicable" in 2018-20).

Labels encode the scheme's own 2-D cross: a dryness family (green / fresh / dry_at_harvest / dried) × a physical form (_with_shell_and_stalk / _with_shell / _in_cob / _in_pods / _grain).

The canonical cross-country vocabulary is declared in lsms_library/data_info.yml the way plot_features.Tenure is — type + prose note + spellings with empty variant lists, carrying the house "countries may extend this list rather than force-fit a local category" stance. _enforce_canonical_spellings applies spelling maps to index levels, not just columns (country.py:3962), which is what makes a Columns: declaration legitimate for an index level.

condition is sentinel-filled (unknown_condition) rather than left NA, because the collapse's groupby(level=...) defaults to dropna=True and silently deletes rows with a null index key.

Verification of the a5aq6b/a5aq6c claim

Checked per wave against the Stata metadata, deciding by the value-label vocabulary, never the variable label. The old comment's claim that "the WB .do's A5aq6b/A5aq6c unit/condition rename is inverted for these actual UNPS files" is true of exactly one file — 2013-14 AGSEC5A. Every other wave labels its variables correctly. The wiring (6c = unit) was right in all cases; only the explanation was overstated. Corrected in place; the full per-wave table is in Uganda/_/CONTENTS.org.

2009-10 and 2010-11 carry no value labels at all; their mapping was confirmed from code distributions (6b's modes are 20/45/24 = the condition scheme's modes; 6c's are 22/1/10 = Plastic Basin / Kg / Sack 100 kg).

Measured (cold, isolated LSMS_DATA_DIR)

wave	dup groups before	after	rows before	rows after	Quantity (identical)
2009-10	670	181	24 997	25 485	310,664,565.8
2010-11	608	81	20 442	20 970	708,149.5
2011-12	653	55	19 552	20 152	1,128,560.3
2013-14	589	61	17 053	17 585	1,098,713.5
2015-16	764	63	18 874	19 576	1,050,367.4
2018-19	0	0	14 194	14 194	1,180,031.7
2019-20	370	18	15 369	15 721	1,175,600.8
total	3 654	459	130 481	133 683	317,005,989.0

Collisions fall 87.4%. Quantity, Quantity_sold (18,134,251.4) and Value_sold (9,066,365,499.8) totals are identical per wave before and after — summing is conservative, so a correct fix moves rows apart without creating or destroying mass.

Residual 459 groups are now all commensurable (same crop, same unit, same condition), so summing them is arithmetically valid — precisely what was not true before. 50 groups' rows differ in harvest_month; ~320 differ only in Quantity; 49 are byte-identical source rows; 37 carry the unknown_condition sentinel and so may still hide a real difference.

Feature('crop_production')

Byte-identical before and after: 250 692 rows, the same 8 kept countries, the same modal index ['country','i','t','v','plot','crop','u','currency'], the same 6 excluded frames [Ethiopia, EthiopiaRHS, Mali, Nigeria, Tanzania, Uganda].

Uganda was already modal-excluded — its shape ['i','t','v','plot','j','u','season'] diverges from the modal ['i','t','v','plot','crop','u'] — so a 7th level changes nothing. crop_production stays out of index_info, per the standing note there that the plot-level ag features need cross-country index NAME harmonisation (plot vs plot_id, crop vs j, across 14 countries) before registration. That is a separate PR.

Tests

tests/test_uganda_crop_condition.py, 10 tests, schema rules read from data_info.yml / data_scheme.yml / categorical_mapping.org, never hardcoded.

Negative control — on the pre-fix tree (stashed working copy, Uganda cache physically cleared): 6 failed, 3 errors, 1 passed, including

test_fresh_and_dry_no_longer_collide: expected the dry (240) and fresh (100)
coffee records to stay separate, got quantities [340.0]
test_condition_is_an_index_level: crop_production index is
['i','t','v','plot','j','u','season']

Post-fix, from a cleared cache: 10 passed. Related suites (test_schema_consistency, test_canonical_spellings, test_table_structure, test_uganda_tables, test_uganda_invariance, test_uganda_v_grain_invariants, test_quantity_kg): 454 passed, 60 skipped.

Deliberately NOT in this PR

Four adjacent defects found while verifying — each moves data, so each needs its own before/after. All four are written up in Uganda/_/CONTENTS.org "Known Issues":

1. 2018-19 AGSEC5B's harvest unit exists but is unwired. a5bq6b has the full 40-label harvest_units vocabulary and 7 041 non-null values, but the colmap sets unit: None, so every 2018-19 season-B row carries u='Unknown'. The long-standing note "2018-19's harvest side carries no unit label" is correct for 5A and wrong for 5B.
2. 2019-20 season B's condition2 slot is dropped entirely. s5bq06a_2/b_2/c_2, labelled "(2019, condition2)", 1 306 non-null conditions, absent from the table. 2018-19's _2 slot is labelled "second season of 2017" and may be a season slot — needs the questionnaire (question: Uganda 2019-20 crop_production silently drops the PART-harvest columns (s5aq06*_11/_21) #647).
3. The dedup collapse silently deletes rows with a null index key — 431 rows / 7 108 132 native units in 2009-10, all with a null plot id. Pre-existing; condition is sentinel-filled precisely so as not to add to it.
4. 2009-10 quantities carry an unstripped 99999 sentinel — that wave's total is ~300× every other wave.

Also out of scope per the issue: the part-harvest columns s5aq06a_11/_21 (#647), any aggregation: YAML key (D1: dead config), and any reducer (D1: a new index level or nothing).

Follow-up the maintainer should weigh

spellings: declares the vocabulary and rewrites known variants, but nothing rejects a value outside it — the precedent for an enforced index-level enumeration is a code constant (transformations.S_VALUES for s), which is core and out of scope for a config-only PR. A follow-up should either add a CONDITION_VALUES constant alongside S_VALUES, or — better — drive an index-level enumeration check from data_info.yml so s stops being special-cased. Recorded in .coder/ledger/323-uganda-crop-condition.md §6.

No cells blessed: no human has read the per-condition numbers in real analysis.

Refs #637, #647.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Su5JKX3wKChyfMAdXrdCTr

[ligon]

Adversarial review — verdict: APPROVE-WITH-NOTES (one comment I'd fix before merge)

I set out to break this and could not break the data path. Every headline number reproduces exactly on an independent cold build, and the two failure modes this class of fix is prone to — zero-by-splitting and invariance by missingness — are both absent from the built table. The findings below are three documentation defects and one test-coverage gap. None of them changes a number in the table.

Method. Isolated LSMS_DATA_DIR (only dvc-cache symlinked in), LSMS_COUNTRIES_ROOT + PYTHONPATH pinned to a detached worktree of b28b5024 and asserted at import (assert '649' in lsms_library.__file__). Uganda cache physically removed between runs, not just LSMS_NO_CACHE=1 (soft for script-path L2-wave parquets). No dvc CLI; all reads via get_dataframe() or the sidecar md5 → local blob → pyreadstat for Stata metadata.

---

Findings

F1 — MEDIUM — the code-99 justification is wrong by ~100×, and it is the premise for a semantic merge

Uganda/_/categorical_mapping.org:858 says code 99 is other_condition — "the more conservative reading; only 3 rows in the whole panel and none after 2011-12".

Measured: 340 rows in the built table carry other_condition; 365 raw source rows carry code 99 alongside a quantity. 219 of the built rows are after 2011-12 — the case the comment says does not exist:

wave	2009-10	2011-12	2013-14	2015-16	2018-19	2019-20
other_condition rows	1	120	32	32	32	123

Why it matters: code 99's label genuinely differs by vintage — verified from the Stata metadata, 'Others' in 2011-12, 'Not Applicable' in 2018-19 and 2019-20, and absent from the label set entirely in 2013-14 / 2015-16 (both ship 19 codes, no 99). The PR merges all three onto one index value. No summing results — t is in the index, so vintages never share a tuple — so this is not a data defect. But it is a semantic conflation whose stated justification is false, and per the CLAUDE.md rule about unevidenced claims in YAML/org comments it will suppress exactly the scrutiny that would catch it. Fix the comment, or split 99 by vintage.

F2 — LOW — data_scheme.yml:152 understates the sentinel rate ~20×, and contradicts the PR's own CONTENTS.org

The comment reads "unknown_condition where the wave records no condition code or an off-scheme one (~1% of 2009-10 and 2010-11 rows…)". Because it covers nulls as well as off-scheme codes, "~1%" is wrong: panel-wide the sentinel is 7,994 / 133,683 = 6.0%, and 2009-10 alone is 5,911 / 25,485 = 23.2%.

"~1%" is correct only for off-scheme codes (I measure 1.1% in 2009-10, 0.9% in 2010-11, 489 rows panel-wide). Worth noting in the PR's favour: CONTENTS.org:390 states the true figures and its per-wave breakdown (5911 / 914 / 712 / 409 / 26 / 9 / 13) matches my independent measurement to the row. So this is an internal inconsistency in the schema comment, not a misunderstanding.

F3 — LOW — "Source Label is the VERBATIM Stata value label" is not true

categorical_mapping.org:852. It is a normalized, cross-vintage representative. Verified diffs against the raw files: en-dash → hyphen throughout 2011-12/2013-14/2015-16; the 2015-16 AGSEC5B mojibake � → hyphen; collapsed double spaces in 2018-19; and four codes whose text comes from a different wave than the row's own —

code	2018-19 raw label	org Source Label
24	Fresh/raw harvested - in pods	Fresh/raw harvested - in pods or shell/husks
32	Dry at harvest - with shell	Dry at harvest - with shell/cob without stalk
42	Dry after additional drying - with shell	Dry after additional drying - with shell and without stalk
99	Not Applicable	Others / Not Applicable

Harmless at runtime — I confirmed _harmonized_codes reads only Code + Preferred Label, and the 3-column table parses to exactly 20 codes with unique labels — but "verbatim" is the wrong word for a provenance column, and the comment contradicts itself two sentences later where it correctly describes the drift. On code 45 specifically ('Dry – grain' → dried_grain, ~25% of all records): the rename is an inference from grid position, not from the literal label. I think it's the right inference — 35 already occupies "Dry at harvest – grain", so 45 must be the additional-drying counterpart — but it is an assertion the label alone does not make, and the org file should say so as plainly as it says the rest.

F4 — MEDIUM — the suite cannot detect a silently unwired condition column

crop_production_for_wave falls back to the sentinel with no warning when the colmap names a column absent from the source:

if cond.get('condition') and cond['condition'] in df5.columns:

I mutated CROP_COLMAPS['2018-19']['A']'s condition to a5aq6b_TYPO, physically cleared the Uganda cache, and reran the suite: all 10 tests passed, while 2018-19 season A went from 20 distinct conditions to unknown_condition on 7,153 / 7,153 rows. The three tests that look like they'd catch it don't:

• test_every_wave_sources_a_condition_column checks the colmap names a column, not that it resolves;
• test_condition_values_are_canonical passes because unknown_condition is canonical;
• test_condition_actually_separates_records is a panel-wide > 1000 threshold that one broken wave-season cannot trip.

This is the "invariance by missingness" mode, and it is live here for a specific reason: 2018-19 legitimately mixes naming conventions ('qty': 's5aq06a_1' next to 'condition': 'a5aq6b' — I checked, both genuinely coexist in that file), so a real typo is indistinguishable from correct config by inspection. Cheap fix: raise or warn in the else branch when a condition column was named but not found, and/or assert a per-wave-season ceiling on the sentinel share.

Related, smaller: test_fresh_and_dry_no_longer_collide carries if sel.empty: pytest.skip(...). It genuinely passes today, but a skip-on-empty escape hatch on the PR's single most important regression test means an id remapping would silently disarm it rather than fail it.

---

What I tried hard to break and could not

1. The unknown_condition catch-all is not the Autre crop defect. This was my main line of attack. Decisive test: rebuild every wave with an identity condition map (code_<n> for every integer, so no two distinct codes can share a label) and diff row counts against the PR's map. If any two distinct source codes were landing on one tuple, the identity build would carry more rows. Result: 133,683 vs 133,683 — zero rows lost, in every single wave. I validated the harness rather than trusting it (asserted the code_* labels actually appeared; my first attempt silently no-op'd and I caught it). So the sentinel absorbs only genuinely-NA codes plus off-scheme codes that never co-occur on a tuple, and the only residual merging it causes is NA+NA — unresolvable without inventing data. The catch-all is non-injective in principle and injective in fact.

2. The 2009-10 / 2010-11 inference is correct — and I tested it in a way that could have come out backwards. Cosine similarity of each unlabelled column's code-frequency vector against the fully-labelled 2011-12 reference:

column	vs CONDITION	vs UNIT	verdict
2009-10 a5aq6b	0.997	0.133	condition
2009-10 a5aq6c	0.149	0.931	unit
2010-11 a5aq6b	0.994	0.128	condition
2010-11 a5aq6c	0.158	0.729	unit

The two reference vectors are near-orthogonal (cos = 0.110), so the test genuinely discriminates rather than returning "similar to everything". Independently, 2009-10's a5aq6c codes behave like physical units — median reported quantity orders exactly as it must: Sack(100kg) 2.0, Sack(120kg) 3.0, Basket(5kg) 13, Bunch(Medium) 30, Kilogram 50, Number of Units 70. The fix is not inverted for those two waves. The same test also confirms 2018-19's unlabelled a5aq6c is a second condition column (0.976 vs 0.127), so preferring the labelled a5aq6b there is right.

3. Conservation is real and is not hiding a deletion. Cold builds of both trees reproduce the PR's table exactly, wave by wave: 130,481 → 133,683 rows; Quantity 317,005,989.0; Quantity_sold 18,134,251.4; Value_sold 9,066,365,499.8 — identical on both sides. I then instrumented the pre-collapse frame and counted null index keys under the 6-level and 7-level keys on the same frame, so the comparison is exact: identical in every wave (2009-10: 431 rows / 7,108,132.2 units, plot only; 2011-12: 43 / 962.5; all others zero). condition contributes exactly zero new null-key deletions. The sentinel fill does what it claims.
One caveat the PR doesn't draw: 2009-10's Quantity total is 99.6–99.8% the unstripped 99999 sentinel (3,169 rows carrying ~316.9M of the 317.0M panel total), so for that wave the conservation check has very little power to detect a real error. The PR flags the sentinel as known issue #4 but doesn't connect it to the weakness of its own headline number.

4. D1 compliance is clean. No reducer added; no aggregation: key anywhere in Uganda's data_scheme.yml. The only groupby/sum tokens in the diff are comments. The pre-existing collapse is unchanged in behaviour — it merely gained a level. This is an identifier fix, which is what D1 asks for.

5. The Feature('crop_production') claim is vacuous — and the PR says so itself. I read all 14 countries' declared index from config: modal is (t, i, plot, crop, u) (8 countries); Uganda's is (t, i, plot, j, u, condition, season), unique. Uganda was already excluded pre-fix at (t,i,plot,j,u,season) — it uses j where the modal uses crop, and carries a season nobody else has. So "byte-identical, 250,692 rows" was never going to be anything else: the cross-country risk was not cleared, only not exercised. Worth stating for the future harmonisation PR: if Uganda were registered in index_info today, condition would be an 8th level the other 13 countries lack, and Feature assembly would have to either drop it — re-summing fresh onto dry in the pooled table, reintroducing this exact bug one layer up — or broadcast NA into it for everyone else. That hazard should be written into the harmonisation issue now, while the reason is fresh. (I did not build Feature across 14 countries cold; this is from config plus the PR's own measurement.)

6. The headline example is real, and the negative control reproduces. Country-level, cold cache:

t       i          plot           j      u  condition     season Quantity harvest_month
2011-12 1033000506 1033000506-1-6 Coffee Kg dried_in_pods A      240.0    6
2011-12 1033000506 1033000506-1-6 Coffee Kg fresh_in_pods A      100.0    6

On the pre-fix tree with the Uganda cache physically removed: 6 failed, 3 errors, 1 passed — exactly as claimed. The single pre-fix pass is test_index_is_unique, a guard rather than a test of the fix. Tests read schema from data_info.yml / data_scheme.yml / categorical_mapping.org; nothing hardcoded, per the house rule. I also confirmed the mechanism the Columns: declaration leans on — _enforce_canonical_spellings does rewrite index levels (country.py:4021-4022, df.rename(index=..., level=col)) — though with every variant list empty it is documentary today, which the PR states plainly.

7. The 2019-20 _1/_2 reading is right, and the 18 residual groups are fully explained. The Stata variable labels read literally 6c. Condition/state of crop harvested (2018, full harvest, condition1) / (condition2) — condition slots, not ordinal harvests. Of 1,382 season-A rows with both slots populated, 133 report the same condition code, and 18 of those also share the unit — which is exactly the 18 residual duplicate groups reported for that wave.

8. The scope calls check out. 2018-19 AGSEC5B's unwired unit is real (7,041 non-null, 40 labels; all 7,041 built rows carry u='Unknown') but cannot interact with this fix — 2018-19 has 0 duplicate groups before and after, so an extra unit level can only keep it at 0. 2019-20 season B's dropped _2 slot is real (1,306 non-null conditions, labelled "(2019, condition2)"); it leaves that wave incomplete, so its residual-collision count will move when wired, but it invalidates nothing here because both sides of the before/after read the same extraction.

---

Summary

The identifier is right, the decode is right, the inference on the two unlabelled waves is right and independently confirmed, mass is conserved, and no new nulls are introduced. My residual-group counts differ slightly from the PR's (426 vs 459 post-fix; 3,611 vs 3,654 pre-fix) purely because I excluded null-plot rows before grouping — same direction, same ~88% reduction, and my harvest_month sub-count lands on 50 exactly as reported.

I'd fix F1's comment before merge (it is the stated basis for a merge on an index level, and it is false), and I'd take F4 as a follow-up since it is cheap and guards the exact failure mode this fix is most exposed to. F2 and F3 are comment corrections.

No cells blessed, correctly — nobody has read per-condition numbers in analysis yet.

[ligon]

Addendum — regression suites confirmed green under the same cold isolated setup.

test_schema_consistency, test_canonical_spellings, test_table_structure, test_uganda_tables, test_uganda_invariance, test_quantity_kg: 288 passed, 67 skipped in 491s, exit 0 — isolated LSMS_DATA_DIR, LSMS_COUNTRIES_ROOT pinned to a detached worktree of b28b5024, Uganda cache physically cleared beforehand. No regressions from the extra index level.

(Slightly smaller counts than the PR's 454/60 because I omitted test_uganda_v_grain_invariants from my selection; nothing failed in the overlap.)

This does not change the verdict above — APPROVE-WITH-NOTES, with F1 (the code-99 comment) worth correcting before merge and F4 (the silent-sentinel test gap) worth a follow-up.

[ligon]

All four findings applied — c8b7e63e

Every number below was re-measured independently before being written down (isolated LSMS_DATA_DIR with only dvc-cache symlinked in, LSMS_COUNTRIES_ROOT pinned to the worktree and asserted, Uganda cache physically removed, no dvc CLI). None of it was copied from the review on trust. All four of your figures reproduce exactly.

---

F1 — corrected, and the merge is kept with a real justification

Your numbers hold to the row. Built table:

wave	2009-10	2010-11	2011-12	2013-14	2015-16	2018-19	2019-20	total
other_condition	1	0	120	32	32	32	123	340

219 after 2011-12. Raw source rows carrying code 99 with a reported measure: 381 (I count 381 rather than your 365 — I kept a row if any of qty / qty_sold / value_sold was present, which is the filter the build itself uses; on the narrower "qty only" reading we'd converge). Vintage semantics confirmed from the Stata metadata: Others in 2011-12, Not Applicable in 2018-19/2019-20, and absent from the label set in 2013-14/2015-16 — both ship 19 codes, no 99, while 32 rows per wave use it.

I did not change behaviour. The merge is kept, and the corrected numbers do not undermine it — but the stated reason had to be replaced, because the old one was false:

1. t is an index level, so a 2011-12 "Others" tuple and a 2019-20 "Not Applicable" tuple can never coincide. Nothing is summed, aggregated or de-duplicated across the vintage boundary; the merge is a naming choice and the vintage stays recoverable from t.
2. 99 is the scheme's own residual slot in every vintage — the code the instrument uses when the dryness × form grid does not apply. other_condition names the slot without asserting which residual reading applies.
3. Splitting would need a third value for 2013-14/2015-16, where no wave says what 99 means — i.e. inventing a label the source does not carry, which is what unknown_condition exists to avoid.

The split you offered as the alternative (other_condition for 2011-12 / not_applicable_condition for 2018-20, sentinel for the rest) is recorded as a follow-up: it moves 340 rows between index values, so it needs its own before/after.

One more inaccuracy in the same paragraph, which you flagged in passing but which the comment still asserted: "The CODES are bit-identical in every wave that carries value labels." They are not — 2013-14 and 2015-16 ship 19, not 20. Corrected.

F2 — corrected

Sentinel share 7,994 / 133,683 = 6.0% panel-wide, 23.2% in 2009-10 (then 4.4 / 3.5 / 2.3 / 0.1 / 0.1 / 0.1%). The "~1%" was the off-scheme rate and only in the two unlabelled waves: 297 = 1.1% of 2009-10, 192 = 0.9% of 2010-11, 492 rows panel-wide (your 489 — same measurement, mine taken pre-collapse). data_scheme.yml now carries the true figures and says explicitly which rate is which.

F3 — corrected, and the four codes are confirmed

Of the 216 (code, wave, season) label pairs across the five labelled waves, 60 match the org column literally and 156 do not. It is the 2018-20 wording with whitespace runs collapsed, except exactly the four you named — 24, 32, 42, 99 — which keep the longer 2009-16 phrasing. Code 99's text (Others / Not Applicable) is a composite that appears in no wave at all.

Your point on code 45 is taken and written down: every labelled wave writes exactly Dry - grain, and dried_grain is an inference from grid position (35 already occupies "Dry at harvest - grain"). Measured coverage 40,411 / 133,683 = 30% of all records, so it is load-bearing and now says so.

F4 — closed, with a mutation proof in both directions

Two guards, because they catch different mistakes.

1. The silent fallback now raises. uganda._require → CropColmapError when a colmap names a column the source lacks. None stays the declared, auditable way to say "this wave records no such column" (2018-19 season A genuinely has no harvest unit). Applied uniformly to every per-slot and id column, not just condition — a mis-typed qty drops a whole slot, which is worse.

Measured before changing it, per your standard: across all 7 waves × 2 seasons × every slot, exactly one named column failed to resolve — and it was a live bug (see below). With that fixed, a cold rebuild is byte-identical: 133,683 rows, Quantity 317,005,989.0, Quantity_sold 18,134,251.4, Value_sold 9,066,365,499.8, intercropped non-null 75,625, sorted frames .equals() → True.

2. Three tests, thresholds set from measurement and with their provenance in the source:

• test_crop_colmap_columns_resolve_in_source — a named column must exist (reads raw Stata, S3-guarded via the house _aws_creds_available() pattern; PR test: one creds guard, and a net for the modules that forget it #648's conftest.py is not on development yet).
• test_sentinel_share_bounded_per_wave_season — ceiling 40%; worst honest cell is 2009-10/A at 25.7%.
• test_condition_varies_within_every_wave_season — floor 10 distinct non-sentinel values; measured minimum 18.

Mutation proof (Uganda cache physically cleared before each run):

mutation	before	after
baseline	10 passed	13 passed
condition: 'a5aq6b_TYPO' (your mutation)	10 passed	1 failed, 4 passed, 8 errors
condition: 's5aq06f_1' — existing but wrong	10 passed	1 failed, 12 passed

The second is the one worth reading. Pointing 2018-19/A's condition at the harvest-month column (a plausible 6f/6b fat-finger) builds cleanly, passes the resolve test, and leaves the sentinel share at 33.7% — under the 40% ceiling. The variety floor is what catches it, at 2 distinct conditions. So a sentinel ceiling alone is not sufficient, and I deliberately did not tune the threshold down to pretend otherwise; that counter-example is recorded next to the constant.

I also took your smaller note: test_fresh_and_dry_no_longer_collide's skip-on-empty is now an assert. And the crop_production fixture re-raises a build failure when S3 credentials are present — without them it still skips, so the data-free CI job is unaffected. Otherwise your own mutation would have produced eight quiet skips, which is the same silent-failure mode the tests exist to close.

---

What the review missed: intercropped is wired to the seed-use question

Found while measuring the blast radius of the strict check — I read the Stata variable labels of every column CROP_COLMAPS[...]['intercrop'] names, not just whether they resolve.

intercropped is built from intercrop.flag with 2 -> True. In every wave that populates the column, that flag points at the seed-use question:

wave	wired flag	its variable label	value labels
2011-12	a4aq3	Did you use any seed or for this crop?	1 Yes, 2 No
2013-14	a4aq16	A4AQ16: Did you use any seed or seedlings...?	1 Yes, 2 No
2015-16	a4aq16	A4aq16: did you use any seed or seedlings...?	1 Yes, 2 No
2018-19	s4aq16	16. Did you use any seed/seedlings?	1 Yes, 2 No
2019-20	s4aq16	16. Did you use any seed/seedlings?	1 Yes, 2 No

The actual cropping-system question is a different column in the same file, present in every wave: a4aq7 "Cropping system" {1: Pure Stand, 2: Inter cropped} in 2009-10/2010-11, a4aq8 / s4aq08 "What type of crop stand was on the plot?" {1: Pure Stand, 2: Mixed Stand} from 2011-12 on.

The two are near-independent — (wired == 2) agrees with (crop-stand == 2) on 48.5 / 48.8 / 52.5 / 51.8 / 52.5% of rows for 2011-12 … 2019-20. A coin flip. Anything using crop_production.intercropped today is using the seed-purchase question.

And the 2010-11 case is exactly the F4 failure mode, live: flag: 'a4aq3' names a column that does not exist in that wave's AGSEC4A (it ships HHID, prcid, pltid, cropID, a4aq7..a4aq14), so the old in df4a.columns guard swallowed it and intercropped was NaN on all 20,970 rows with nothing said. Set to None here — a provable no-op that makes the config state what the build already did, and lets the strict check apply uniformly. Rewiring to the real question adds and moves data (it would newly populate 2009-10 and 2010-11), so it is filed as a Known Issue rather than folded into a comment-and-tests commit.

Your F4 mutation only touched condition; this one was sitting in the same dict.

---

Verification

tests/test_uganda_crop_condition.py: 13 passed from a physically cleared cache (was 10). Full sweep — test_schema_consistency, test_canonical_spellings, test_table_structure, test_uganda_tables, test_uganda_invariance, test_uganda_v_grain_invariants, test_quantity_kg: 627 passed, 60 skipped, exit 0 (10m33s, cold). The only warnings are the pre-existing Uganda cluster_features GrainCollapseWarnings, unrelated to this table.

Constraints held: no aggregation: key; no edit under lsms_library/*.py outside countries/Uganda/; data_info.yml untouched this round; no dvc CLI. Still no cells blessed — nobody has read per-condition numbers in analysis.

Full write-ups in Uganda/_/CONTENTS.org ("Code 99 is merged across vintages", "Source Label is a normalized representative", "A mis-wired colmap must fail loudly", and two new Known Issues) and .coder/ledger/323-uganda-crop-condition.md §Phase 4.

[ligon]

Addendum — 12346608: prior art I missed on the first pass

I applied F1–F3 having read only the sections of Uganda/_/CONTENTS.org that PR #649 itself added. I've now read the file end to end. No contradiction — nothing pre-existing asserts anything about code 99's incidence, the sentinel rate, or the unlabelled 2009-10/2010-11 codes, so none of the corrected numbers overwrote a recorded decision (I checked the LOGBOOK/status entries too; the only one is on 2019-20 panel weights, and there are no WAITING caveats). But one piece of genuine prior art was missed, and it reframes F3 from a wording error into a structural one.

The convention already exists — one column per wave

CONTENTS.org §"Unit handling for food_acquired" pre-dates the condition work and was untouched by b28b5024:

The u table also retains the per-wave variant columns (2005-06, 2009-10, …, 2019-20) that document the raw label appearing in each wave's questionnaire. These extra columns are unused at runtime but preserve the cross-wave provenance previously captured in the retired _/unitlabels.py file.

So categorical_mapping.org currently holds three shapes for one job:

table	shape	per-wave drift
u	Code, Preferred Label, one col per wave	representable
harvest_units	Code, Preferred Label	not recorded
harvest_conditions	Code, Preferred Label, Source Label	lossy

The single Source Label column is not merely where the "verbatim" claim went wrong — it is why it could. One cell cannot hold two vintages' wordings, so codes 24 / 32 / 42 silently took the 2009-16 text and 99 became a hand-made composite that appears in no wave. The u table has no such failure mode: it keeps each wave's string in its own column, typos and all — Small cup wuth handle(Akendo) survives verbatim across six waves. That is what "verbatim" is supposed to mean, and it is what a single column structurally cannot deliver.

Recommended follow-up (documented, not done)

Give harvest_conditions the u table's shape: replace Source Label with one column per labelled wave (2011-12, 2013-14, 2015-16, 2018-19, 2019-20; 2009-10 and 2010-11 stay blank, since they ship no value labels — exactly as the u table leaves blanks). It is runtime-inert (get_categorical_mapping reads only Code + Preferred Label), and the 216 measured label pairs already recorded in this PR are precisely the data needed to populate it.

I did not do it here: you scoped F3 to a prose correction, and a table reshape is your call rather than a side effect of one. Say the word and it's a small, mechanical change.

Written up in Uganda/_/CONTENTS.org (new subsection under the Source Label correction), pointed at from the categorical_mapping.org comment so the next reader hits it at the table rather than having to find this thread, and recorded in .coder/ledger/323-uganda-crop-condition.md.

Docs only, no behaviour change. tests/test_uganda_crop_condition.py: 13 passed from a fresh isolated LSMS_DATA_DIR (cold rebuild confirmed — no reuse of the earlier run's cache).