v0.14.0

v0.14.0

Drafted 2026-09-16.

A release about caches and trees that were trusted without being checked,
under two invariants that #918 is careful to keep apart. The first is
laundering: a make recipe's output was accepted as current because it
existed, then re-stamped with a fresh hash, so the evidence that anything was
wrong was destroyed on the first read -- a hashless country parquet graded
legacy and was trusted once (#809), and a stale one survived make doing
nothing and was served and re-stamped (#866). The second is ownership: the
runtime wrote into the package tree, which on a shared install is not
writable, so a plain read died with PermissionError (#914, #746, #831 read
path). Around those: a country-qualified unit moves out of KNOWN_METRIC
into config (#919), a labels= defect that summed 146 unlabelled foods into
one blank bucket (#925), a metric rung on the crop kg ladder (#929), and a
GhanaLSS release inside the release -- own production valued at the farmgate
price in four waves, and five farm tables wired for the GLSS1/GLSS2 panel.
The wave-to-country food-label crosswalk that GhanaLSS had carried unapplied
is applied as well. On top of it, GhanaLSS gets a curated
Aggregate Label column, so labels='Aggregate' resolves there for the first
time (#940).

Upgrading: every cache re-warms, and #866 is no longer an exception

Both parquet tiers invalidate, for every table in every country, and
three independent changes each say so. #918's edit is inside
_aggregate_wave_data, "@build_transform() with no tables=, so every
build fingerprint moves and the corpus re-warms cold" -- measured,
food_acquired d7c1c7b0376785ec -> 5e295682b582044b, household_roster 7a87dbd16447b8e9 -> c476d3f90477b20c. #924's load_from_waves is nested in
the same function, and because Wave.grab_data folds
build_transforms_fingerprint, "every L2-wave parquet goes stale too.
Nothing harmonized is reusable." #921 hashes categorical_mapping/u.org
(gorg:), so "every Country._table_cache_hash moves and the corpus
re-warms". All three were authorised, #918 and #921 explicitly at @ligon's
direction.

The country-scoped re-warms are subsumed by that but worth naming, because
they say which files move a hash: #917's edits to nine country Makefiles move
cmake= for every table in Burkina_Faso, Ethiopia, EthiopiaRHS, GhanaLSS,
Mali, Niger, Senegal, Tanzania and Uganda "for a change that alters no build
output"; #916 (country module plus four wave scripts) and #934
(data_scheme.yml is hashed whole) each re-warm GhanaLSS whole, including
the 5.26M-row food_acquired.
#935 re-warms GhanaLSS for the same reason: categorical_mapping.org is in
every GhanaLSS table's input hash.
Zero-cost, measured: #917's data_access.py change ("three
probed build_transforms_fingerprint values byte-identical"), #922 ("no
data_access symbol reaches any build fingerprint"), and #927 ("0 of 10
build_transforms_fingerprint and 0 of 12 Country._table_cache_hash values
moved"). #930 reports no cache figure.

The two previous releases said "#866 is still open -- run lsms-library cache clear --country {C} first." That advice is withdrawn for the ordinary
read path (under LSMS_BUILD_BACKEND=make, #924 says, try_make still
serves an existing var/ parquet for 53 no-rule pairs; see "Filed, not
fixed"). #924 removes a stale country parquet before the make/script fallback
runs, so the nine cells that used to serve a stale frame now rebuild:

form	cells
no rule	Nigeria crop_production, livestock, people_last7days, plot_inputs, plot_labor
rule, mtime-skipped	Ethiopia nutrition, EthiopiaRHS community_prices, GhanaLSS nutrition, Nigeria anthropometry

Two costs, stated in #924: "three Makefile execution failed ... No rule to make target warnings per affected build, where today there are none",
deliberately not suppressed; and "assume_cache_fresh=True stops being a
recovery lever for those nine cells". A genuine pre-v0.8.0 country parquet now
rebuilds instead of migrating (#918) -- "nil in practice (0 of 376), and
v0.8.0 is five minor versions back".

Dependency change: CFEDemands>=0.10.0 (was >=0.8.2; #940). The
aggregation pipeline scores with Regression.score_w, which earlier versions
do not have, and 0.10.0 changed score_w's min_goods default -- the
pipeline now passes min_goods explicitly everywhere (e70b8df on #937,
carried into #940) so no result depends on a library default. A shared venv
on 0.8.2 needs pip install -U CFEDemands before running anything under
SkunkWorks/aggregation/. The library itself imports cfe.df_utils
(transformations.py, at import time) and cfe.regression lazily
(demands.py); both resolve unchanged under 0.10.0 (checked), so a venv on
0.8.2 still imports the library -- it only cannot run the pipeline.

Behaviour changes you will see in returned data

• GhanaLSS own production is valued at the farmgate price in four waves
  (#916). GLSS4-GLSS7 record a Quantity and the household's own reported
  farmgate Price for home-produced food but never a total, so Expenditure
  was left NaN and food_expenditures(basis='total') returned <NA> for own
  production in every wave but the two 1980s ones. Expenditure = Quantity * Price now lands as GhanaLSS::food_acquired::8h-farmgate, stamped on
  exactly the 605,333 rows it computed: own-production shares 30.5% (1998-99),
  17.1% (2005-06), 27.8% (2012-13), 28.8% (2016-17). Not comparable with
  12b-fortnight, which asks a consumer price. 1991-92 is deliberately not
  wired -- its dominant unit is All (93,349 rows), "the whole harvest",
  and it stays <NA>. Served Price is identical before and after in all
  seven waves; food_prices(units='kgvalue') now values produced rows it
  used to return NaN on.

• food_prices includes derived valuations, by rule (#923). "A derived
  value is a first-class value: no served table may exclude one for being
  derived" (@ligon, 2026-09-14). GhanaLSS food_prices moves from 407,478
  rows in one wave to 568,346 in four (kgvalue) and 1,513,769 to 1,674,641
  (unitvalue); unitprice (189,551) and units (1,693,174) are asserted
  unchanged on purpose, since a valuation must not move the reported-price
  or quantity modes. This does not reverse "empty food prices is better than
  a made up constant": that ruling is about a fabrication, and a registered
  derivation names its function, cites its inputs and stamps every row.

• A hashless country parquet is rebuilt, not trusted (#809). At the
  country tier "hashless" does not mean "old": a make recipe's
  to_parquet('../var/x.parquet') passes no cache_hash=, so a parquet
  written moments ago graded legacy, was served, and was re-stamped with the
  current hash -- "0.6 s is far too fast to be a rebuild". The wave tier keeps
  legacy deliberately (script-path wave parquets are hashless by design).

• A stale country parquet is removed before the fallback (#866).
  Reproduced on Nigeria/livestock: 23,196 rows poisoned to 5, served in
  0.8 s with no rebuild, re-stamped with the correct hash. The eviction is
  placed only before the country-level fallback, because 13 Make rules across
  12 countries consume a var/*.parquet as a prerequisite and deleting one
  at the read gate is not a local operation.

• A country-qualified unit is declared in config (#919). quintal was
  seeded into KNOWN_METRIC at 100 kg, which "beats every country-level
  factor unconditionally"; it is the metric centner in Ethiopia and 100 pounds
  (45.36 kg) in Central America. A u_kg table in categorical_mapping/u.org
  now carries it, keyed on the canonical unit and overridable per country. No
  Ethiopian kilogram moved (100.0 both ways); 13,977 rows moved from the
  shipped rung to metric, and removing the seed alone would have served
  0.682 / 1.024 kg on Ethiopia's 16 food_acquired Quintal rows. Still
  wrong and stated: Panama's 23 centner rows serve an inferred 22.09 /
  42.34 kg against its own 45.359.

• labels=<col> no longer renames an unlabelled food to '' (#925).
  _relabel_j guarded with .dropna(), and blank org cells parse as empty
  strings, so every food the requested column did not label was renamed to
  '' and, under reaggregate=True, summed into one unnamed j bucket:
  146 foods in GhanaLSS labels='1987-88', 90 in 1991-92, 24 under FCT Code; 17 (country, column) pairs corpus-wide. Expenditure was conserved
  in every case, which is why it was silent.

• The crop kg ladder distinguishes metric from inferred (#929). A
  factor read off a unit label is no longer bucketed with one guessed by
  price-ratio inference. A pure relabelling: 0 of 85,519 Ethiopian
  kg_per_unit values differ, Harvest_kg is identical, and inferred is 0
  in both splits -- the inference serves Ethiopia not one crop row, now
  pinned.

• GhanaLSS gains five farm tables for the GLSS1/GLSS2 panel (#934).
  crop_production (37,280 rows), plot_inputs (11,282), livestock
  (6,407), labor (36,722) and plot_labor (6,042), 1987-88 and 1988-89
  only, all is_this_feature_sane ok. crop_production has no u level
  and no harvest quantity (Part B asks what was sold); Value_sold and
  HerdValue are registered derivations; Cost is the corpus's first
  input-spend column, and Transport/Containers/Storage are the cost of
  selling, a quarter of recorded spend. The Farm-level sentinel on
  plot_id / j states a survey's grain positively instead of dropping the
  level. PurchaseValue and Cost are registered monetary (0b8a1b532),
  so the FX/PPP conversion layer no longer walks past them.

• GhanaLSS food labels sit on the country axis (#935). The country
  harmonize_food table records each round's label against the canonical
  one, and nothing applied it: 45 of 225 served j labels were the wave
  vocabulary, 14% of rows and ~36% of GLSS2 expenditure. The crosswalk now
  runs in all seven wave scripts and an unmapped label raises. Seven
  premature collapses are split (Rice into Rice / Rice (imported) /
  Rice (local), likewise smoked fish, sugar, cocoa powder, macaroni, potato,
  gari), and the label defects are corrected: Cocoyam Leaves held Coffee
  in 1998-99 (8,935 rows served the wrong FCT code), Cassava (flour) is gari
  in every wave, Cola Nut and Kola Nut were one food (19,076 rows).
  Delivered: 0 labels off the axis, 0 duplicate tuples, 215 = 215 distinct
  labels, Expenditure per wave identical in all seven. nutrition is not
  rebuilt or inspected and will change; Oil (palm)/Shea Butter is
  deliberately left without an FCT code (4,239 rows drop out of nutrition).
  The Aggregate Label column itself is still not designed. The four
  GhanaLSS pins #923 had just moved move again for the split rows:
  kgvalue 568,346 -> 568,624, unitvalue 1,674,641 -> 1,676,093,
  unitprice 189,551 -> 189,656, units 1,693,174 -> 1,694,623
  (measured on the release-candidate tree, 8766f882c).

• GhanaLSS has a curated Aggregate Label column (#940, superseding
  #937). labels='Aggregate' resolves for GhanaLSS instead of raising
  LabelUnavailableError. Every one of the 215 country labels #935 delivers
  carries an Aggregate (219 rows, 0 blank), 209 groups: 205 singletons and
  four semantic groups that pass the CFE beta-spread gate
  max(beta) - min(beta) < 1.348 * sigma_min (one fit per group, seeded
  bootstrap sigma; SkunkWorks/aggregation/ghana_gate.py): Sugar (cubed/granulated) (ratio 0.96), Malt Drinks (0.80), Macaroni/Spaghetti
  (0.27 on the full wave), Millet/Sorghum = {Millet/Sorghum, Millet, Guinea
  Corn/Sorghum} (0.62; 0.12 / 0.96 / 0.12 in 2012-13 / 2005-06 / 1998-99).
  Every label #935 renamed or split was decided at the gate rather than
  inherited: rice (3.7), gari (3.1), smoked fish (2.0), palm oil/shea butter
  (3.6), cocoa (2.5) and bread/sugar bread (3.5) fail and stay separate;
  goat/mutton fails 2012-13 (1.08); beer/pito fails three waves; #937's
  Kapla/Tilapia group fails the primary wave (1.17) and is dropped;
  margarine/butter, potato/sweet potato and the other pairs the gate cannot
  fit ship as singletons. Two borderline calls, Goat/Mutton and Crab/Shrimp
  (primary 1.21 against 0.31 on the full wave), ship conservatively as
  singletons and are recorded in .coder/ledger/ghanalss-aggregate-recuration.md
  for a human. The 47 rows #937 had appended for "API-delivered" names are
  gone: they were the wave labels #935's crosswalk now maps. Held-out
  validation re-run on the new axis: coverage unchanged to four decimals.
  Stated, not resolved (#939): every partition was scored at
  min_goods=1 on 83-88-good fits, i.e. a household can be scored from one
  observed good; whether the partition ranking is stable in min_goods is a
  sweep nobody has run. nutrition.py now selects wave columns by the
  YYYY-YY pattern so the new column is not read as a survey round.

Framework and cache

• The runtime does not write into countries_root() (#914, #746, #831
  read path). The nine countries shipping panel_ids.json carried a Make
  rule that regenerated it whenever panel_ids.py was newer -- and "on a pip
  install which of the two is newer is decided by the millisecond the wheel
  was unpacked": on the real 0.13.0 wheel, 5 of 9 landed with the script
  ~1 ms ahead. The blast radius was every table, since _finalize_result
  calls id_walk(df, self.updated_ids) on every read. The JSON keeps its
  place as a prerequisite and loses its rule; regeneration is make panel-ids. #746 was the same trigger reported six months earlier, with a
  different downstream exception. get_data_file also materialised every
  fetched raw file into the package tree (400 sidecar-beside copies; 33 with
  no sidecar silently shadowing the DVC blob); read-path fetches now land
  under data_root(), acquisition stays the one sanctioned exception. The
  other two #831 bullets remain open.
• The panel crosswalk is a committed source: design A (#894, convention
  pinned). Malawi and Nigeria are named in DESIGN_B_PENDING rather than
  grandfathered; converting them is #894 item 2 and stays open.
• get_data_file takes the sidecar fast path (#763, via #922). It
  fetched through DVCFileSystem, whose exists and get_file both walk
  the index over ~10k sidecars. It now uses the same _ensure_dvc_pulled /
  _dvc_cache_path pair as get_dataframe: 49.0 s to 0.4 s on the same call,
  122x, "a floor, not a refutation of the issue's 1,599 s". #917 had claimed
  this fix and did not carry the commit; #922 re-opened it unchanged.
• ll.countries() cannot be clobbered by importing the subpackage
  (730f0a775, in #916). lsms_library/countries/ is a namespace package
  imported lazily, so the rebind happened after __init__ returned and
  ll.countries became a module -- the #883 trap for a third name, pointing
  the other way. Reachable from DerivationRecord.resolve(); reproduced as
  5 failed, 39 passed. A property on the module's class now pins it.
• DerivationRecord.resolve() returns the callable (#916); the stored
  field stays a string because json.dumps of a callable raises and a
  callable compares by identity after importlib.reload.
• tests/test_panel_ids_persistence.py (56) reproduces the trigger by
  os.utime; tests/test_gh866_stale_country_parquet_not_laundered.py
  poisons content as well as hash and asserts served.equals(ref); the #809
  test that pinned trust-once is rewritten stating both sides, not deleted.

Documentation

• CLAUDE.md: the countries_root()-is-read-only invariant with its three
  issues; the tier split for hashless parquets; #866 and the correction that
  LSMS_BUILD_BACKEND=make "does still READ the country parquet"; the
  first-class-derived-value rule and its reason.
• .sucoder/handoff.org (#933): development was never red -- #930 was
  built against a shared mirror 15 commits behind GitHub, and #919 had
  already fixed the same root cause by the opposite route. The check is
  git rev-list --left-right --count development...origin/development.
• GhanaLSS/_/categorical_mapping.org's Aggregate-Label passage corrected
  (#927); the #928 config-only survey of categorical key columns
  (slurm_logs/categorical_key_vocabulary/); #934's design notes in
  slurm_logs/2026-09-16_panel_estimation/.
• #935's measurements and each decision with its author in
  slurm_logs/ghanalss_aggregate_labels/.
• #940's gate runs, held-out validation and every grouping decision in
  SkunkWorks/aggregation/ghanalss/ and
  .coder/ledger/ghanalss-aggregate-recuration.md; the aggregate-labels skill
  now says GhanaLSS curates the column.

Filed, not fixed

#926 (refs, #927); #928 (categorical key column);
#931 (harvest_kg_factors does not receive unit_kg, so the crop ladder is
blind to declared u_kg rows -- which is what keeps Ethiopia's external check
non-tautological) and #932 (#930, #933); the LSMS_BUILD_BACKEND=make and
wave-level laundering routes (#924, own issues); #831's remaining two bullets
and #894 item 2 (#917); Panama's centner and the migration of universal
spellings out of KNOWN_METRIC (#921).

Issues closed

• #746 -- panel_ids.json regenerated on upgrade; the trigger removed (#917).
• #763 -- get_data_file walked the DVC index instead of taking the sidecar fast path (#922).
• #809 -- a hashless country parquet was trusted once and re-stamped (#918).
• #866 -- a stale country parquet survived make and was laundered (#924).
• #914 -- PermissionError from a Make rule regenerating a committed source (#917).
• #919 -- quintal hardcoded in KNOWN_METRIC at 100 kg (#921).
• #925 -- labels=<col> renamed unlabelled foods to '' and summed them (#927).
• #929 -- the crop kg ladder counted read and guessed factors together (#930).