Skip to content

v0.15.0

Choose a tag to compare

@ligon ligon released this 15 Oct 16:18
· 3066 commits to master since this release
cb44e35

v0.15.0

Released 2026-10-15 from cb44e3522a2426ba03431ca9a218cc63fe03f6c9.
Available on PyPI and as
GitHub Release v0.15.0.
This is the full change set since v0.14.0, including work merged in September.

Final implementation validation passed at 3914d8804. The maintainer
approved Uganda PR #967 after reviewing its nutrition accounting and documented
assumptions; PR #1014 repairs the assets compiler. Combined credentialed CI
passed 4,927 tests with no failures, and the final focused Uganda selection
passed 158 tests with no skips. The final corpus pass checked all 549 declared
requests. The known source gaps and independently explained cold/warm
differences remain explicit below; validation is not blanket scientific
approval of every output.

New survey coverage and discovery

GhanaAHIES adds the Ghana Annual Household Income and Expenditure Survey,
2022Q1–2024Q4.
Eight tables cover all twelve quarters: sample,
household_roster, individual_education, food_security, housing,
cluster_features, people_last7days and anthropometry.
household_characteristics derives from the roster. The three annual source
folders retain quarterly t values and weights, with a stable household key
constructed from cluster and household number; the source hhid changes by
quarter and is not used as the panel identity. PR #944,
PR #950.

The survey's consumption, expenditure, income and asset modules are not
available as complete tables in the distributed files, so the library does
not claim those features. Source caveats remain visible: 2023Q3 household
weights have a half-scale anomaly, and some quarters contain nonnegative
anthropometric values that may encode nonmeasurement. They are recorded rather
than corrected without a decision. GPS is not supplied. See the
country documentation.

Wave discovery now represents multiple holdings and catalog relationships.
Provenance can record several catalog IDs, constituent coverage, evidenced
aliases and derived releases without treating them as additional acquired
microdata. Tanzania's four constituent NPS releases can report covered by
the 2008–15 holding, and Nigeria's derived catalog entry can report derived;
the existing Boolean local remains false for those statuses. Existing scalar
catalog-ID and constructor behavior is retained. Missing or incomplete
provenance remains uncertain. PR #992.

Corrections to returned data

  • Niger 2014–15 households include EXTENSION in their identity. This
    restores 59 distinct households and their person and agriculture records.
    Exact all-nines missing-value codes are decoded before arithmetic. Crop sales
    carry Unit_sold, repeated sale exports are removed before summing, and
    selected food/region spelling variants are harmonized. Seed-slot collisions,
    orphan harvest records and the cross-wave split-off namespace caveat remain
    documented. PR #993,
    #679,
    #865,
    #824.
  • Nigeria W1/W2 crop sales retain their recorded sale unit. This does not
    supply the missing plot link for W3–W5 household-crop sales or recover other
    buyer channels. W5 now supplies cluster geography in both 2023Q3 and 2024Q1,
    using household-specific keys for EA=0 movers. The repository's v01 extract
    lacks GPS; the available v02 geovars are a separate acquisition tracked in
    #978. The old claim that
    the survey supplies no W5 geovariables has been corrected.
    PR #994,
    PR #995.
  • Malawi preserves the sale's own shelled/unshelled basis and distinguishes
    boiled groundnuts.
    Registered derivations disclose the sale-basis ladder,
    including price-supported overrides and a separate no-reference outcome;
    null-unit sales remain in basis references. This does not resolve every
    crop-to-food identity or processing comparison. PR #966.
  • Uganda reads the second season-B harvest-condition slot for 2018–19 and
    2019–20.
    Food acquisition distinguishes source-confirmed money-only
    rows, unknown units, residual codes and the recorded fish-unit label instead
    of converting all 7,359 old blank-unit rows through fabricated factors.
    Quantities and expenditures at acquisition grain are preserved; inferred kg
    and nutrition outputs change. PR #972,
    PR #967.
  • Uganda's Makefile assets compiler preserves the item-level observations.
    It now concatenates intact wave frames and applies the existing household
    crosswalk, producing 116,808 Value rows on (t,i,j). The previous script
    expanded them to 790,688 rows and removed the item key from the index.
    Exact private replay matches the canonical raw and served API outputs;
    the API's existing 4,237 duplicates after item-label mapping remain unchanged.
    The general builder-authority and divergence-detection decision remains
    #788.
    PR #1014.
  • Burkina Faso 2014 preserves missing quantity. The four visits remain
    separate seven-day recalls; only passage 1 has a quantity grid. Exactly
    449,033 manufactured zero quantities become missing, while expenditure,
    other quantities and row coverage remain unchanged in the cold comparison.
    Food-label crosswalk parsing is repaired. Price diagnostics distinguish
    money-only groups, with the limitation that an all-null quantity column
    alone cannot prove questionnaire coverage. PR #981.
  • Peru 1994 gains sample geography with consistent padded cluster IDs.
    All 3,623 sample households and 19,285 roster records join the 364 clusters.
    The export's constant-one _weight_ is not presented as a survey weight;
    the other Peru sample waves remain unwired. Panama 1997 code 77
    (valor) no longer supplies 7.70 as a physical quantity, and the raw
    expenditure total is preserved. #700,
    #777,
    PR #996.
  • GhanaLSS food scripts use script-relative imports, its live 1998–99
    Price reducer is corrected, and curated C1-control punctuation in labels is
    repaired. Region spelling and unit-label repairs also cover Ethiopia, Mali,
    Malawi, Niger, Guinea-Bissau, Burkina Faso and Senegal. The explicit-metric
    parser now reads fractions such as 1/2 kg as fractions.
    PR #947,
    PR #952,
    PR #949,
    PR #971,
    PR #963.

Kinship cleanup removes adjudicated unreachable keys and duplicate entries,
and fixes source labels that reached lookup as empty strings or literal
Nan. Reachable spelling-variant folding remains open in
#974; no new relationship
decomposition is implied by the cleanup.

Portability, readers and diagnostics

Script-path builds can invoke the wave script when GNU make is unavailable;
Windows conda setup, UTF-8 file handling, installed-wheel identity checks and
manual CI dispatch support accompany this path. Wave builds run serially where
process forking is unavailable, including native Windows. Linux cold tests produced
matching Uganda shocks/assets results with and without make. The installed
native Windows installed-package cold-build/comparison job passed on the final
implementation 3914d8804: 14,457 shocks rows and 116,808 assets rows across
all eight Uganda waves, with the expected content digests. Its separate smoke
selection passed 327 tests with one skip.
Windows validation.
Importing the package no longer eagerly imports GitPython.
#964,
#968.

The reader dispatches SAS XPORT by magic bytes, including the verified Peru
SSP files, and rejects contradictory or undecodable claimed Stata files before
a permissive text fallback can turn binary bytes into a misleading empty
frame. Peru 1990 table configuration remains work to do. Reader fallbacks and
ID crosswalk reads no longer use cwd-rooted dvc.api.open; JSON cache reads no
longer invoke make. #699,
#712,
#810,
#970.

  • labels= normalizes label-column names, reports unmatched labels through
    warnings and labels_unmatched, supports strict refusal, and rejects numeric
    targets. Ambiguous one-to-many targets, per-country aggregate curation and
    carrying diagnostics through Feature remain open in
    #787.
  • Monetary scale diagnostics compare neighboring waves and warn at the
    approved fivefold threshold without changing the coverage tier. Currency
    boundaries and support checks remain explicit; warnings are investigative
    signals, not proof of corruption. Persistent shifts, long gaps and other
    blind spots remain in #830.
  • Country-name typos raise early with suggestions. Existing but unregistered
    directories remain the unresolved half of
    #915. Categorical mapping
    strips string values without nulling mixed-column numbers, and illustrative
    housing tables no longer enter the live mappings.
  • Reducer conflict reports survive cold/warm cache reads, including explicit
    rounds inside a multi-round folder. Panel diagnostics check roster-cache
    wave coverage before treating missing targets as a data defect.
  • cache clear reports in-tree source files and preserves tracked configuration
    inputs. The sidecar-less in-tree read policy remains open in
    #831.
  • Canonical food deduplication and Panama scripts use vectorized
    sum(min_count=1). Summed floats can differ by a few ULP; the regression
    bound is four. The remaining reducer sweep is
    #905.

Upgrade and validation notes

The automatic publishing run
passed. The downloaded PyPI wheel contains exactly the 11,751 package files
from the tagged source. Rebuilding the published source archive without Git
produced identical contents for all 11,756 wheel members, including metadata;
the compressed archives differ only in DEFLATE encoding and associated
size/offset fields. A fresh PyPI installation passed dependency checks,
20 public API/configuration checks and a separate ordinary Uganda assets call
on a private warm-cache copy: 116,808 rows with the exact expected content,
all eight waves and no cache changes. This last check is a warm installed
data check; the credentialed cold Windows builds are recorded separately.
Publication and installation evidence.

Expect a cold rebuild of both harmonized parquet tiers across the corpus.
The build and reader changes share a corpus-wide fingerprint scope. L1 source
blobs remain reusable. A successful warm call is not validation of the new
build. The integrated cold rebuild covered all 549 declared requests across
38 countries: 530 succeeded, with two unavailable Armenian source requests,
eleven unavailable Nepal source requests and six declared but unwired Peru
requests reported as errors. No other build errors occurred. Two Uganda
country caches, earnings and fct, lose their metadata when sibling scripts
rewrite them; ordinary reads correctly treat them as stale under #809.
The original outputs remain preserved, and follow-up audits use independent
copies. Corpus evidence.

The final ordinary pass at 3914d8804 checked the same 549 requests, with
530 successes and the same 19 errors. All 28 Uganda served results match the
reviewed cold corpus. Trailing ordinary fct and earnings calls restored
their hashes and coverage; however, rebuilding fct runs nutrition.py,
which writes both outputs and leaves nutrition hashless. The final country
snapshot therefore has 423 fresh files and one hashless nutrition file, with
unchanged raw values. That file was excluded from promotion. Its old shared
counterpart was absent, so no retirement was needed. Its next ordinary API
call rebuilds it. This preserves the existing sibling-write limitation in
#788; repeated reads cannot
be presented as a way to make every sibling permanently fresh.

Shared-cache promotion verified 1,518 files: 423 country files with current
hashes and full built-wave coverage, 632 fresh wave files, 461 script-wave
files with verified build provenance, and two verified panel-ID companions.
The transaction created 153 paths, replaced 1,215 with recoverable backups,
and retained 150 identical files. L1 and both private evidence roots remain
unchanged; no files were deleted. Final cache validation.

The minimum ligonlibrary version is now 0.2.3. Python remains >=3.11,
pandas remains >=3.0, and the existing CFEDemands >=0.10.0 requirement is
unchanged. poetry.lock is now tracked; the prior environment was archived
before the fresh resolve. Dependency versions are not part of the data-cache
fingerprint, so the environment change also needs cold validation. Development
installation and image-building guards use --no-root and explicit checkout
identity. Multi-node venv ownership and its documentation remain separate
unfinished work.

CI avoids duplicate branch/PR runs, supports merge queues and manual credentialed
validation, and skips known unavailable data at collection rather than waiting
for network failure. Unexpected runtime credential skips warn. Structural tests
enumerate declared tables and make absent cache coverage visible; they do not
turn an unbuilt table into evidence of correctness. The requested sequential
cold selection for #941
passed 451 tests with no skips at ed1414fb9, including all EHCVM delivery
assertions. Corrected credentialed run
38697245468
completed with 4,919 passes and three Uganda failures: two assets structural
checks and the nutrition fixture comparison. All 163 EHCVM module outcomes
passed; the historical Derivation/inkind failure did not recur. The separate
collect-only test environment leak (#1010)
is fixed by PR #1011. This does not establish the historical #941 cause,
so #941 stays open. Final combined-source run
38746123119
passed on 3914d8804: 4,927 passed, 2,768 skipped, four expected failures and
one existing non-strict unexpected pass
, with no failures, in 3,204.33 seconds.
This executes the assets repair and approved nutrition fixture together.
Its unit job passed 3,293 tests with 4,407 skips; Windows
smoke passed 327 tests with one skip, and native script builds passed as above.

The final master run
38759333750
passed on the tagged release commit with the same test counts: 4,927 data
passes, 2,768 skips, four expected failures and one existing non-strict
unexpected pass, in 3,137.91 seconds. The formerly failing Uganda assets and
nutrition assertions actually passed. Unit, Windows smoke, installed native
Windows cold builds, strict documentation, remote reachability and DVC lock
checks also passed; no lock refresh or master change occurred.
Master CI evidence.

The actual release target on master cb44e3522 passed 7,376 tests with
zero failures or errors
, with 314 skips, nine expected failures and one
existing non-strict unexpected pass in 1,024.34 seconds. It used eight
loadfile workers on an independent complete-cache copy. No credential-guard
skips occurred. The expected failures are four existing weight expectations
and five documented tenure defects. One Guyana cached sanity check skipped
after a read retry exhausted; the same assertion passed separately on an
isolated physical copy. Its original cause remains unknown and the original
314-skip count is retained. Release-target review.

The full local credentialed suite at ec1a59767 passed 4,951 tests, with
2,725 skips, four expected failures and one existing non-strict unexpected
pass. None of the skips were credential skips; 2,580 were absent-cache cases.
Their follow-up against the cold-built corpus, with ten additional helper and
identity checks, produced 2,416 passes, 168 classified skips, five existing
tenure-label xfails and the then-held nutrition mismatch. Fifteen tests
require the unavailable independent Uganda replication project. The nutrition
fixture comparison skipped in that full run; the approved #967 follow-up
subsequently executed all 53 baseline entries successfully. Some legacy test preconditions require stored derived parquets that the
current API does not write; these and obsolete index/dtype assertions remain
coverage debt in #1006.
Full-suite evidence,
cache-presence follow-up.

Two helper tests now isolate and restore the cached data-root setting even on
exceptional teardown. Credentialed CI pins build and numerical-library thread
counts and uses a 600-second per-test timeout while retaining sequential test
execution and the same test selection. Neither change establishes the cause
of the historical #941 failure. PR #1004,
PR #1005.

The complete Feature sweep covered 39 features, all 549 declared country/table
pairs, 38 baseline assemblies and 17 configured variants. It retained 19 known
source/wiring errors and five existing tenure-vocabulary failures in Albania,
Cambodia, Kosovo, Tajikistan and Timor-Leste. Its diagnostic warnings remain
visible; completion is not a blanket scientific approval of every warning.
Older saved audits are partial and lack matched source/cache provenance. Their
comparison and the reviewed source/value changes revealed no unexplained
release-specific warning category, but do not prove every diagnostic historical.
Comparison limits.
Feature audit evidence.

Region-market food variants still warn and exclude GhanaSPS and Panama: both
have cluster IDs in sample(), but neither exposes sample.Region or declares
the cluster_features geography fallback. The relevant code and declarations
are unchanged since v0.14.0. GhanaSPS remains covered by open
#140; Panama is tracked in
#1009.

Mali and Tanzania still lose valid interview dates on all-wave warm reads.
Mali's complete raw cache and cold API contain 24,939 rows; its warm API serves
12,194, dropping all 12,745 dates from the two EHCVM waves. Tanzania's cold API
serves 22,427 rows; its warm API serves 16,534, dropping 5,893 valid dates from
2019–20 and 2020–21. Tanzania also has six originally null dates, excluded in
both cases. Whole-column datetime coercion infers one format from mixed string
representations, creates missing dates, and the finalizer removes those rows.
Independent replay exactly reproduces both losses. The relevant source route
is unchanged since v0.14.0, and the earlier report already named these waves;
the October 14 cold-only claim that they no longer reproduced was incorrect.
#798 remains open. Complete
raw parquets are preserved separately from incomplete served outputs.

On a complete warm cache, ordinary per-wave calls followed by concatenation
recover the exact cold values, keys, order and dtypes in both countries:

import pandas as pd
import lsms_library as ll

country = ll.Country('Mali', preload_panel_ids=False)  # also verified for Tanzania
dates = pd.concat([country.interview_date(waves=[wave])
                   for wave in country.waves])

The ten calls were verified on a separate physical cache copy without bypasses
or file changes. This is a measured workaround, not a repair of the all-wave API.

Two other cold/warm differences come from serialization. Liberia's 2,787
interview dates reload as milliseconds instead of seconds; all timestamps,
keys and missingness remain exact. Mali's roster retains all 220,732 rows and
43,823 missing MonthsSpent values, but numeric month scalars become strings
on reload. Numeric values and all other columns remain exact; direct arithmetic
and type-sensitive consumers still see an interface difference. The existing
EHCVM binary-to-12/0 proxy also conflicts with the country's duration guidance;
this validation neither changes nor scientifically approves that proxy.
Serialization evidence.

GhanaLSS still loses null-key rows during Feature assembly. On the rebuilt
corpus, Feature('food_acquired')(['GhanaLSS']) deletes 13,472 rows with a
missing cluster or unit key; the country-native output retains them. This is
separate from the legitimate aggregation of visits. Independent replay
reproduced the dropped rows and conserved the retained additive values.
The mechanism is unchanged since v0.14.0; exact counts on rebuilt historical
inputs were not measured. The earlier reporting issues did not fix this
country residual. The measured loss is tracked in
#1008.
Independent review.

Uganda nutrition was approved on October 15. The historical fixture was
23,801 x 15 with hash fc85fc2e…; pre-#967 development is the same
shape with hash 1131b358…, and this candidate is 23,777 x 15 with hash
12d25f6e…. Historical replay exactly reproduces the first two hashes and
attributes their entire difference to four #850 commits already shipped before
v0.14.0. Their 18.7208% Energy reduction is historical baseline reconciliation,
not a change introduced by v0.15.0.

The merged #967 increment has independent raw-row and nutrient arithmetic
accounting: inferred quantity falls by 849.299850578 kg, 24 household-wave rows
lose their last convertible quantity, and 60 retained nutrient vectors change.
Arithmetic replay does not adjudicate the inherited litre-as-kg and nominal
container assumptions. It also exposed an older food-composition source-note ambiguity:
sweet-potato row 4003 supplies vitamin A at 225 mcg/100 g, while its source note
says vitamin A and carotenoids were assumed zero. An adjacent, more specific
note records beta-carotene derived from three yellow-flesh varieties in Kidmose
et al. (2007), supporting a nonzero value. The follow-up is to verify that cited
derivation and reconcile the generic note; it is not evidence that the density
should become zero. No density or numeric source value was changed. The
maintainer approved #967 after reviewing these documented limitations. Only the
nutrition fixture's shape and fingerprint were updated; all 53 recorded
baseline entries now match, and the focused units/kg/invariance selection
passed 127 tests with no skips. This acceptance does not independently validate
nominal volume conversions or change an FCT numeric value. See the
historical accounting
and dated approval and validation.

The historical #837 early
cluster-reduction proposal was reviewed and rejected. Current development
already retains all 333 Ethiopia W1 GPS fixes; the proposal would instead blank
364 Ethiopia GPS cells and 119 Tanzania attributes and bypass the existing
household-grain audit. Source behavior stays unchanged. The country notes now
correct the false inference that Ethiopia W1's missing-versus-recorded GPS
conflicts were administrative disagreements. Remaining source/scientific
questions stay open under #837.

Detailed issue dispositions and reconciliation evidence are in
RELEASE_ISSUES.org.