Feature('sample') oddities: int_t leak, Rural spellings, China/Azerbaijan weight quality

[ligon]

Sibling of #161 (Uganda cluster_features) and #162 (cross-country stray i).
Feature('sample') is structurally much healthier — no duplicate indices, no stray
i column — but it surfaces a separate set of issues worth tracking.

Summary

489,406 rows, 32 countries, clean (country, i, t) MultiIndex, no duplicates.

Issues

1. Stray int_t column (severity: medium)

int_t (dtype float64, 467,703 / 489,406 non-null) leaks into the sample output
despite belonging in interview_date. Almost certainly a wave-level data_info.yml
declaring int_t under a sample: stanza when it should be under
interview_date:. Grep across all countries to find the offenders.

2. China weight entirely null

China's sample DataFrame has 787 rows, all 787 with weight = <NA>. The
optional: true override applied in the v0.7.0 release makes this schema-valid,
but the data gap is real and worth a sanity check — is China's sample frame
genuinely unweighted, or did a mapping silently drop the column?

3. Rural spelling inconsistencies (canonical spellings gaps)

Rural        275290
Urban        178609
rural          5790   # lowercase variant — should map to 'Rural'
urban          4090   # lowercase variant — should map to 'Urban'
0              2263   # numeric code — some country has 0 for a category
Semi-urban     1578   # genuine third category
Informal         83   # genuine fourth category

Roughly 10K rows with case variants that the canonical spellings dict in
lsms_library/data_info.yml should catch but doesn't. 0 (2,263 rows) is a
numeric code that needs an explicit mapping: block in whichever country's wave
YAML surfaces it.

Suggested fixes:

• Add rural, urban (lowercase) to the spellings lists for Rural/Urban in
  data_info.yml. Alternatively, apply .str.title() normalization in
  _enforce_canonical_spellings.
• Locate the country producing '0' for Rural and map it.
• Semi-urban / Informal are legitimate — should stay in the canonical set
  rather than being coerced to Rural/Urban.

4. Azerbaijan weights look like a fallback

Azerbaijan 1995 sample has 2,016 rows but the weight column has only 3
distinct values (min 0.694, max 1.258). That's an extremely uniform
distribution and may indicate a default / fallback weight rather than true
survey weights. Worth verifying against the raw microdata (PP.dta per
the data_info.yml).

5. Resource-handle leak (minor / benign)

lsms_library/country.py:831 appears to load YAML without with context, leaving
file handles unclosed. Not behaviorally broken but flagged by the audit pipeline.

Out of scope

Not a release blocker — v0.7.0 has shipped. Queueing here for a v0.7.1 or
later cleanup pass.

[ligon]

int_t leak: likely RESOLVED on development by 13761a4 (int_t: Int_t in Rejected Spellings) + be2cb46 / 08939f4 (country-level and wave-level data_scheme.yml/data_info.yml renames). Needs a confirming Feature('sample') probe — the rewrite fires at the Country API boundary but raw parquets may still carry int_t until rebuilt.

Rural lowercase spellings: not yet fixed. Two-line data_info.yml addition (rural: Rural, urban: Urban in the spellings dict for the cluster_features.Rural column). Queued as a quick win.

'0' Rural numeric code: not yet localised — needs a grep across wave-level data_info.yml files for a country emitting '0' as the Rural value. Separate small fix.

Keeping open for the two Rural items.

[ligon]

Closing — items 1, 2, 3 fixed; items 4, 5, 6 split off.

What landed

• Item 1 (int_t leak) → 9959b9f3. Direct-read of 31 cached
  sample parquets confirmed Azerbaijan 1995 and Timor-Leste 2001 were
  the only offenders. Both declared int_t: <col> inside sample: df_cover: myvars: while their data_scheme.yml has no
  interview_date table — orphaned keys. Removed.
• Item 2 (Rural/Urban lowercase spellings) → 0c6847a4. Added
  spellings dict for canonical Rural / Urban values in
  lsms_library/data_info.yml (catches rural / urban / RURAL /
  URBAN variants for Malawi, Tajikistan, and Uganda/2019-20 at API
  time).
• Item 3 (Uganda '0' Rural, 2,263 rows) → 9959b9f3. Uganda/2005-06
  sample stanza was a bare Rural: urban with no mapping block; added
  string-keyed mapping ('0': Rural, urban: Urban). The raw column
  is string-typed — numeric keys would not fire (cf. CLAUDE.md).

Cache caveat

Cached parquets at ~/.local/share/lsms_library/{Country}/var/sample.parquet
still hold pre-fix values. The fixes only land in Feature('sample')
output after those caches rebuild. Either wait for regular cache
invalidation or run LSMS_NO_CACHE=1 / lsms-library cache clear for
Uganda, Azerbaijan, and Timor-Leste to force a rebuild.

Split off

• Feature('sample'): China weight column 100% null (787/787 rows) #174 — China weights 100% null (likely CONTENTS.org doc fix,
  per handoff note).
• Feature('sample'): Azerbaijan weights look like a fallback (3 distinct values, 2,016 rows) #175 — Azerbaijan fallback-looking weights (needs raw-microdata
  verification).
• country.py:831 leaks file handle — yaml.safe_load(open()) without context manager #176 — country.py:831 bare open() in yaml.safe_load (minor
  hygiene).