feat: principled auto-aggregation script for harmonize_food's Aggregate Label column

[ligon]

Motivation

The Aggregate Label column on each country's harmonize_food table (added in 094a585 / 96cd278 for Malawi) groups Preferred Labels into demand-coherent buckets so that Country.food_expenditures(labels='Aggregate', reaggregate=True) returns a smaller index whose elasticities can be interpreted as if estimated on the underlying items. Today the column is hand-curated. It would be useful to automate this against a falsifiable criterion -- without losing the type-coherence judgements that currently keep us from putting Eggs - Boiled (Vendor) (β=0.51) in the same bucket as Maize Ufa Refined (Fine Flour) (β=0.51).

Proposed criterion (sketch)

Two pieces interact. A bucket should:

1. Pass a β-similarity test at low confidence (50% has been the working threshold). Concretely: re-running CFE on Aggregate labels yields one β per bucket; that β, suitably normalised (e.g. beta.mean() = 1 over both label sets), should not be significantly different from each constituent's individual β estimated under Preferred labels.

2. Be type-coherent. A bucket of items with similar β but unrelated culinary type would pass (1) but mislead any downstream analyst. This is the piece that resists pure automation -- some form of food-label semantic similarity is required.

There's also a bonus criterion related to sample weight w that Ethan flagged: aggregation should aim to lift each bucket clear of cfe.regression.prepare_data's min_obs=30 and min_prop_items=0.1 thresholds, weighted by household sampling weights w. The current spread heuristic ignores w entirely.

Implementation sketch

Three components.

A. β-spread / weight-aware test

Inputs: per-PL (β, σ, n_eff) from a Preferred-labels CFE run, plus household-level w from Country.sample(). Output: for any candidate bucket, whether the projected aggregate β stays within the 50%-CI envelope of every constituent.

Approximate the aggregate β as the weighted mean (Σ wᵢ nᵢ βᵢ) / (Σ wᵢ nᵢ); the constituent test simplifies to a closed-form comparison. No CFE rerun needed at proposal time, only at validation.

B. Type-coherence judgement

The piece that needs domain knowledge. Three plausible options:

1. LLM-in-the-loop: prompt with each candidate merge (e.g., {Eggs - Boiled (Vendor): 0.51, Maize Ufa Refined: 0.51} -> would these two food items be treated as substitutes by a typical Malawian household?). Cheap to call, but loose.
2. Pre-existing food taxonomies: FoodOn ontology, USDA FNDDS food groupings, ECVMA/EHCVM official food categories -- map Preferred Labels onto these and require shared parent within some depth.
3. Curator approval: surface the script's proposals (B-pass + A-pass merges) for a human to accept/reject, then learn the country-specific overrides.

(2) is portable across countries once the mapping is built; (1) is fast to bootstrap; (3) is the safety valve.

C. Greedy clustering driver

Given the β-spread test (A) and type-coherence judgement (B), an obvious algorithm is:

• Start from one bucket per Preferred Label.
• Repeatedly pick the pair of buckets that (i) passes both (A) and (B) and (ii) has the smallest β distance, merge them.
• Stop when no pair can merge.

That tends to produce locally-optimal aggregations. Could be improved with a global optimiser, but greedy is a fine MVP.

Scope of the script

Per country. Inputs: categorical_mapping.org (Preferred labels), a CFE β/σ result on Preferred labels, Country.sample() for w and n_eff. Output: a proposed Aggregate Label column that the maintainer reviews and commits.

Could be invoked from lsms_library CLI, e.g. lsms-library propose-aggregate-labels --country Malawi.

Related

• Hand-curated baseline: 094a585 (Malawi original 40 buckets) and 96cd278 (Malawi tightened to 56 buckets after β-spread audit).
• Cross-country relevance: every country with a harmonize_food table will benefit -- Uganda, Tanzania, Ethiopia, Niger, Burkina Faso, Mali, Senegal etc. The β-spread test is country-specific (each survey has its own β estimates) but the type-coherence layer (B-2 / B-3) is reusable.
• GH Design: food_acquired canonical columns (+ cascade to food_prices, food_quantities) #169 / Malawi: harmonize_food wave-labels (and cluster_features districts) drift from raw post-capitalize() data #216 / Malawi: cluster_features['District'] has 6 spurious naming variants for the same admin areas #217 -- earlier Malawi label drift work.
• GH format_id silently truncates non-ID idxvars at first period (affects every country using get_categorical_mapping) #222 -- format_id workspace bug in df_data_grabber that affected harmonize_food key handling.

Acceptance

• Script produces a 'proposed Aggregate Label column' that, when applied to a country's harmonize_food, results in a bucket scheme where the within-bucket β-spread test passes at 50% confidence for all multi-item buckets.
• Documented in .claude/skills/add-feature/food-acquired/SKILL.md (or a new sibling skill) with the propose -> review -> commit workflow.
• At least one country (Malawi or Uganda) has its Aggregate column regenerated by the script, with the resulting buckets within 1-2 splits of the hand-curated baseline.