Skip to content

Import-entry unit family P1: ledger ingest of official trade margins (#615) - #620

Open
MaxGhenis wants to merge 28 commits into
mainfrom
import-entry-margins-p1
Open

Import-entry unit family P1: ledger ingest of official trade margins (#615)#620
MaxGhenis wants to merge 28 commits into
mainfrom
import-entry-margins-p1

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Import-entry unit family P1: ledger ingest of official trade margins

First phase of #615 — the synthetic import-entry unit family. This PR is the ledger leg: official Census/CBP series admitted as ledger-contract facts with full retrieval provenance. No entries are generated here (that is P2, stacked on this).

Source switch (2026-08-05)

P1's primary source is the Census monthly bulk imports database (IMDB), not the International Trade API. Census publishes the complete U.S. Imports of Merchandise database monthly as public no-auth ZIPs:

https://www.census.gov/trade/downloads/{YEAR}/Merch/im_m/IMDB{YY}{MM}.ZIP

(URL pattern verified live 2026-08-05: all 18 archives 2025-01→2026-06 HEAD-checked 200, 118–292 MB each, ~3.3 GB total.) Eighteen GETs replace thousands of 500-prone per-chapter API queries — the API's biggest chapters (84/85/87) time out server-side after ~150 s and need recursive prefix-splitting to materialize at all. The bulk archives carry strictly more detail, and each one ships with the publisher's own control-total files and record layouts, so reconciliation runs against the archive's own published totals.

The API ingest stays in the tree as the independent cross-check leg (census_imports.py, including the prefix-split machinery): the same series fetched over a second official channel, compared cell-for-cell on a stratified sample. Two official channels agreeing is stronger provenance than either alone. The bulk path does not depend on any API code.

Reference implementation for the file geometry: Yale Budget Lab's tariff model parses these exact ZIPs (src/io/build_import_weights.R, HS10 × country consumption imports); its six parsed field positions match the layouts shipped inside the archives (Documentation/IMP_DETL.lay), which are what imdb_bulk.py transcribes.

What's in the box

packages/populace-build/src/populace/build/us_runtime/us_trade/:

  • imdb_bulk.py — the primary ingest. Streams each monthly archive's fixed-width IMP_DETL detail (HTS-10 × Schedule C country × country subcode × district of entry × district of unlading × rate provision) with 22 monthly measures: customs value (con_val_mo), dutiable value, calculated duty, charges, CIF value, both unit quantities, card counts, and air/vessel/containerized-vessel transport splits (value/weight/charges). Every row's embedded statistical period must equal the archive month (a mislabeled row fails the ingest, never silently drops). Reconciliation is exact-integer against the publisher's own in-archive control totals on three independent axes: by country (IMP_CTY), by commodity (IMP_COMM, also the source of units of quantity), and by district of entry (IMP_DE) — any difference fails the build.
  • census_imports.py — the API cross-check leg (unchanged from the lane-G implementation: byte-cached responses, sha-256 manifests, key elision, DET/CGP handling, publisher - totals, prefix splitting for the giant chapters).
  • census_country_bridge.py — the Schedule C → ISO-2 bridge vendored verbatim from axiom-oracles reference/us-tariff-panel/census_iso_bridge.csv (241 rows), hash-pinned and verified at load. Fail-closed: an unmapped country code aborts the assembly. Both legs bridge through the same table.
  • cbp_entry_stats.py — CBP fiscal-year entry-summary anchors (total/informal entry summaries, import value, duty collected) parsed from archived bytes of the public CBP Trade Statistics page; only exact-integer cells become facts. In-progress fiscal years are encoded as fiscal-year-to-date record sets with an explicit as-of endpoint parsed from the page's own "updated as of" note (…fytd.fy2026_as_of_2026_07_27, period.type = fiscal_year_to_date with start/as_of); only completed fiscal years mint fiscal_year facts. The as-of parse fails closed (round 3): the scan is scoped to the text between the entry-summaries table and the next table — the live page carries three "updated as of" notes and the Trade Remedy table's differently-dated one must never become this table's endpoint — and any as-of wording the strict pattern cannot read, conflicting dates, or a missing table note while as-of wording exists elsewhere on the page raises instead of falling back to the retrieval date. The retrieval-date fallback survives only for pages with no as-of wording at all, and the fact builder independently refuses stats carrying a note without a parsed date. Still the only public entry-count series located; the P2 size distribution stays an explicit assumption anchored to these totals.
  • import_entry_facts.py — consumer-artifact emission (policyengine_ledger.consumer_artifact.v1 / ledger.consumer_fact.v1) at five grains now: national, chapter, country, chapter × country, and district of entry (new, from the publisher's own district table, carrying its own IMP_DE source leg — round 3: district values come from the IMP_DE control file, and both source_table members say so; the margins feed keeps its IMP_DETL identity). Source identity is channel-parameterized (FactSourceLeg): record-set ids and populace-namespace fact keys are identical whichever channel supplied the bytes (P3 selector stability), while each row's source block records the actual retrieval channel, file, and sha. Customs value carries concept_alignment to the composed tariff spine's #input.customs_value (exact, Census definition quoted); calculated duty aligns to the total ad-valorem rate as related, never gating (Synthetic import-entry unit family: ledger-margin-generated trade entries, engine-runnable #615).

Plus tools/build_us_import_entry_margins.py (end-to-end bulk build: verify/adopt archives → parse → reconcile → staged, atomically published margins/totals/district parquets + per-month full-detail parquet + consumer artifact + build report; publication is windowless on every steady path — a single-syscall directory exchange (renamex_np(RENAME_SWAP) on macOS, renameat2(RENAME_EXCHANGE) on Linux) where the filesystem offers one, and a symlink layout elsewhere (the public name is a symlink retargeted by one atomic rename of a prepared sibling link, so it never vacates on any POSIX filesystem; round 4 replaced round 3's windowed two-rename fallback). Every destructive step is preceded by a durable fsynced recovery marker — written before the exchange too, so a set displaced by a crash mid-cleanup is reclaimed — and a publisher lockfile refuses concurrent builds toward the same destination, taking over only dead same-host holders) and tools/crosscheck_us_import_margins_api.py (the stratified, gated API comparison), and 103 trade tests (bulk ingest 35 incl. a synthetic in-test archive builder, a rename spy proving the public path resolves across every publisher rename, and subprocess kills at the retarget, inside the legacy migration, and between exchange and cleanup; archive-fidelity goldens 13, facts 26, crosscheck 17, API leg 12 — parse, reconciliation-failure, duplicate-control-key, off-period, truncation, bad-country, adoption, atomic-publication interruption/recovery, lock refusal/takeover, FYTD, as-of fail-closed mutations incl. HTML entities and impossible dates, golden-layout machine-checks, write-spied cache reuse, and offline CLI subprocess paths).

What the bulk source upgrades (previously documented gaps)

  1. District splits — lane G recorded "district/mode splits: not free at this grain; skipped." The archives carry district of entry and unlading in the detail grain. This PR emits the publisher's district-of-entry table as a parquet artifact + feed facts (census_intltrade.imports_district_entry.*), reconciled exactly against the detail.
  2. Mode/transport splits — air, vessel, and containerized-vessel value/weight/charges now ride the margins and detail parquets as published columns.

What the bulk source does and does not ground (all source-verified; corrected in review round 1)

  • No postal/informal/entry-type marker. Rate-provision codes classify free vs dutiable status only (census.gov/foreign-trade/reference/rpcodes.html — codes 00–19 free variants, 61–79 dutiable variants). The transport splits are air/vessel/containerized. is_postal_shipment therefore remains a documented P2 assumption.
  • cty_subco is the published Country SubCode — the special trade-agreement / preference-program claim (0 = no program, A = GSP, S = USMCA, …; census.gov/foreign-trade/reference/codes/csc.html). Carried through the detail parquet as an analytic dimension; aggregated over for margins. (Round-1 correction: originally described as unenumerated.)
  • cards_mo is defined: "Number of Detailed Records, Current Month" (the published import detail file structure). It counts detail records, not entry summaries, so it is still never used as an entry-count anchor — but it now also serves as a reconciled control measure that protects every detail row with current-month activity. (Round-1 correction: originally described as undefined.)
  • District 70 is Schedule D "LOW-VALUED IMPORTS AND EXPORTS" — the publisher's own low-valued-shipments aggregation, present in all 18 months of the district margins (~$2.0–3.4 B/month customs value, zero calculated duty). Not postal, not an entry-type flag, but a real published low-value split the earlier draft wrongly implied did not exist. (Round-1 correction.)

Review round 1 (sol, adversarial): REQUEST-CHANGES → hardening landed

The round-1 review found real weaknesses, all addressed in c05884de: reconciliation expanded from 4–6 measures to every measure each control file publishes (22 commodity / 18 country / 18 district) plus two-directional key-set equality; fact provenance made fail-closed (months without manifest coverage refuse to mint, empty retrieval manifests refuse to write, channel-correct observed_measure, bridge hash pinned in the artifact generator); CBP's volatile "as of" note moved out of identity dimensions; CBP semantic keys made fiscal-year-invariant; district facts keyed on code only; the cross-check tool now gates on API-leg reconciliation failures and publisher-total drift with an offline test battery; the build CLI streams months (bounded memory) and clears stale outputs; the fixture's full-width row makes one-character colspec shifts fail reconciliation; and three overclaimed "no published definition" statements were corrected against the sources (previous section). The checkpoint string-dtype environment-stability fix (45089b27) repairs a latent #614-class defect the new pyarrow dependency surfaced in every venv.

Structure facts established empirically (recorded, tested)

  • The monthly archives are year-to-date cell unions: a cell active in any earlier month of the statistical year persists with all-zero monthly measures (2025-12: 3.51 M detail rows, ~0.81 M with monthly activity). The margins table keeps one row per cell with monthly activity (matching the API leg's semantics); the full published union rides the per-month detail parquet.
  • Every archive in the window is a single-period snapshot (all rows carry the archive's statistical period — scanned across all 18 files) and measures are explicit integers in every row (zero blank measure cells in a full-file scan), which is what the fail-closed parse assumes.
  • 2026 archives name members *.TXT (2025: *.txt); member lookup is case-insensitive and requires exactly one match.
  • Prior-year archives are re-generated by Census with revised statistics (the 2025-01 archive's inner files are dated 2026-03), i.e. the bulk channel serves the current revision like the API does; the cross-check leg tests this directly.

Decisions to note

  1. Producer identity / repo home (unchanged from lane G): facts are populace-minted in a populace_us_trade.* key namespace — not an export of a PolicyEngine/ledger build — and every row's extraction_method plus the artifact generator block say so. The ledger repo's declarative source-package harness hand-enumerates selector rows per fact and cannot express a 10⁵–10⁶-cell series; if it grows a bulk contract, these facts relocate by reproducing the same rows there.
  2. Channel-invariant fact identity. record_set_id stays census_intltrade.imports_hs10.* under the source switch — the statistical series and publisher are unchanged; the channel is provenance, not identity. Semantic fact keys are therefore stable across lane G's API pulls, this build, and any future channel.
  3. Volume split. The JSONL feed carries the five coarse grains; the full HTS-10 × country × month margins and the complete per-month publication-grain detail ride parquet artifacts (runtime inputs, sha-pinned in the build report — not committed, household-feed convention).
  4. USITC DataWeb: still no API token (documented gap); Census calculated-duty/dutiable-value series cover the duties validation leg from official data. CBP entry counts: FY national totals only.

Build evidence (full 18-month window, built at the hardened ingest)

  • Window 2025-01 → 2026-06 (18 monthly archives, ~3.3 GB, each sha256-recorded with retrieval timestamps).
  • 38,841,305 detail rows parsed → 3,070,851 margin cells (HTS10 × country × month with monthly activity), 325,524 publisher commodity-total rows, 792 district rows; 19,224 distinct HTS10, 233 countries (all bridged to ISO-2, zero unmapped), 44 districts.
  • Reconciliation: 0 failures across all 18 months under the full gates — every published measure on all three control axes (22 commodity / 18 country / 18 district) plus two-directional key-set equality and duplicate-control-key rejection, exact-integer. Machine-readable per-month evidence is committed in-branch (below).
  • 302,318 consumer facts (national + chapter + country + chapter × country + 1,584 district facts + 4 CBP anchors, the FY2026 ones FYTD-labeled); artifact round-trips populace's loader against facts_sha256 and the manifest hash.
  • Determinism: four independent full builds across the review rounds produced byte-identical margins/totals/district parquets and per-month detail parquets (identical sha256s below — the round-3 build reproduced every data artifact byte-for-byte through all seven review changes); the facts file changed only through the review rounds' designed provenance changes (round 3: request-time CBP retrieved_at, IMP_DE district source tables; current sha below). The round-3 build re-archived the live CBP page: its boilerplate bytes differ from the round-2 capture but the entry-summaries table and as-of note are byte-identical — same four exact cells, same fy2026_as_of_2026_07_27 record set, no fact-identity drift.
  • Plausibility: monthly national customs value $250–340 B; effective calculated-duty rate 2.2 % (2025-01) rising to ~10.5 % (fall 2025), easing to ~6.8 % (2026) — the known tariff trajectory.
margins_hts10_country_month.parquet  f7f88f5824112d1a9c6beaaf4addc9b15f795f6ba6d6e4447e72957731c29607
census_totals_hts10_month.parquet    62f9fd431e86c14525c4e30fb3456670e5591c8e9720e9395ddec17edc4480d8
district_entry_month.parquet         5d508ba6cd8c719567060dcde5885ad58beb654563cf1df0254e9230edd3f652
consumer_facts.jsonl                 7cd155161ebb77ea4b5145a1d5f61c5e28a85345ea0a18dad729a34afa8179f8
consumer manifest.json               75dc57a69f44731718078bfc2bdfbed926eff14903fa90f25fdf35f2dbd73da0

Committed evidence pack (docs/evidence/us-import-entry-margins/)

The run's claims are reviewable and recomputable in-branch, not prose: build_report.json (per-artifact sha256s incl. all 18 per-month detail parquets and per-month reconciliation-evidence hashes), source_manifest.jsonl (18 archives + the archived CBP page: URL, sha256, size, retrieval timestamps), reconciliation/period=YYYY-MM.json × 18 (per-axis key sets, duplicate-key verdicts, per-measure compared/matched cell counts with both sides' integer totals), and crosscheck_api_report.json (below). Chain of custody: the golden pack's archive sha256s equal the source-manifest rows for both golden months; heavyweight artifacts (3.3 GB of archives, ~1 GB of parquet) stay outside the repo with every byte pinned here.

Cross-check evidence (API leg, gated — exit 0)

Stratified sample of 16 (month × chapter) pairs covering the 500-prone giants (84/85/87), small and mid chapters, both statistical years, and both window endpoints (2025-01, 2026-06): 108,977 HTS10 × country cells compared exact-integer on all four dollar measures — 0 mismatches, 0 publisher-total mismatches, 0 API-leg internal reconciliation failures, 0 absent-totals pairs. The report pins both input parquets by sha256 (equal to the build report's) and every pair's fetched API retrieval manifest. The gate fails on any mismatch, missing counterparty, or an empty comparison.

One comparator defect surfaced by the live run and fixed (ce6753da): the API serves YTD-union rows — after January it returns cells (and - totals) with all-zero month measures for pairs active earlier in the statistical year, which the bulk detail legitimately omits. Zero on one channel with silence on the other is agreement, not divergence; these are now counted per side in the report (45,220 zero-union cells; 427 = 427 zero-carrier totals per side across the sample — the two channels carry identical carrier sets), while one-sided cells or totals carrying any nonzero measure still gate.

Round 3 hardened the comparator's predicates and re-ran the gated check against the round-3 build (exit 0, same 108,977 cells, 0 everything): activity is one predicate on both legs — any nonzero dollar measure OR any published nonzero quantity — so a zero-dollar quantity-bearing total can never be misfiled as a zero carrier (F3), and an active pair with no API totals slice gates even when the bulk totals slice is empty too — the totals leg can never pass by comparing nothing (F4); both r2 probes are committed tests. The committed report also carries repaired retrieval provenance (F7): the pre-fix code had rewritten every cached split marker's retrieved_at on each reuse, so the cached values corresponded to no actual request; the stale markers were deleted and the overloads re-observed live, meaning every timestamp in the committed report is the moment of a real HTTP event (markers from this run's live probes; split-response descendants keep their original cached retrieval timestamps).

Review round 2 (sol, adversarial, on the filed PR): REQUEST-CHANGES → all findings closed

  1. Archive fidelity (F1) — committed an 84 K golden pack: the four official .lay layout members × both golden months verbatim plus feature-selected raw records with expected literals, sha-bound to the download manifest; a machine-check test re-derives the colspecs from the official .lay bytes, and raw records run through the real parser (8c6f6e39).
  2. Atomic publication (F2, critical) — the build now stages to a temp directory and rename-swaps; a failure test with a pre-populated output dir proves byte-identical survival of prior outputs, and a success test proves stale-file removal (fcab9332).
  3. Reconciliation evidence + duplicate control keys (F3) — machine-readable per-month reconciliation records committed in-branch; duplicate control keys now reject per axis, including sol's byte-identical-duplicate repro (fcab9332, 575a4833).
  4. FYTD (F4, critical) — partial fiscal years no longer mint completed-annual facts (see cbp_entry_stats.py above; tests re-encoded, plus complete-year/disjointness/fallback tests) (468d0337).
  5. Provenance (F5)observed_measure is channel-correct for bulk facts; CBP source_file names the actual archived filename; verified_at/retrieved_at split (adopted archives without a download manifest record no retrieved_at rather than a wrong one); the artifact writer refuses manifest entries without retrieval timestamps; the build report hashes the consumer manifest (fcab9332).
  6. Bridge pinning (F6) — provenance cites the immutable axiom-oracles commit URL (9310d19b…), the vendored CSV is verified byte-identical to that commit, and bridge_provenance.json is vendored, pinned, and cross-bound in the loader (8c6f6e39).
  7. Crosscheck gates + evidence (F7) — absent API totals and empty comparisons now gate (exit 1); the report carries input parquet sha256s and per-pair API retrieval manifests; the sample covers the 2026-06 endpoint; live gated run committed (previous section) (fcab9332, ce6753da).
  8. CI + lint + changelog (F8, blocking) — the two 3.14 full-suite failures were Canonicalize string dtypes at every pool serialization boundary (#578: run-7 publication failure) #614's raw-HDFStore assertions demanding storage pandas fixed-format HDF does not persist; tests re-pinned to the persistable contract with an option-pinned exact form and a pyarrow-default regression test (c1bc2de5); ruff I001 ×2 fixed; changelog fragments added.

Nits: measure counts corrected here to 22/18/18, window-endpoint sample now includes 2026-06, test count corrected (74 then; 91 after round 3; 103 after round 4), the district grain now rides ALL_IMPORT_ENTRY_FACT_GRAINS beside the four margins grains (documented: it is emitted from the publisher's district table, not the margins), and strict key-shape gates added (hts10 exactly \d{10}, country/district code shapes).

Review round 3 (sol, adversarial): REQUEST-CHANGES → all seven findings closed

  1. Atomic publication had a crash window (F1, high) — the double rename let readers observe ENOENT between the renames, and a kill inside the window stranded the previous publication under .previous-*. Publication now exchanges staging and destination in one syscall (renamex_np(RENAME_SWAP) / renameat2(RENAME_EXCHANGE); the test suite asserts the swap wherever the filesystem offers it, verified live on the build machine's APFS), so the published path exists at every instant. Filesystems without an exchange fall back to a two-rename swap guarded by a durable recovery marker; _recover_interrupted_publication rolls any interruption forward (staged set intact — it was complete by construction) or back (previous publication restored) at the next build. The interruption test kills the fallback swap between its two renames with os._exit(9) in a subprocess and proves recovery republishes the staged set; in-process second-rename failure restores the previous publication immediately (1cb72f85).
  2. As-of parsing could fail open (F2, high) — unrecognized note wording parsed as "no note", after which the fact builder substituted the retrieval date; rewording the note to a single fiscal year silently recreated the partial-FY bug. The parse now fails closed (scoped scan, unreadable-wording/conflicting-dates/displaced-note raises — details in the cbp_entry_stats.py bullet above); the single-fiscal-year wording itself now parses, so the r2 probe yields correctly-labeled FYTD facts, which is the committed mutation test. Developing this against the real archived page immediately caught the Trade Remedy table's unparseable "All programs updated as of:" note — the fail-closed design had to be scoped to the entry table's own note to be honest (130b92b9).
  3. Totals activity ignored quantities (F3, high) — one shared activity mask (dollars OR published nonzero quantities) now serves the cell leg and both totals legs; sol's zero-dollar quantity-7 probe is a test both ways (absent API slice → api_totals_absent; present slice lacking the commodity → one-sided total mismatch) (e2f0b928).
  4. The totals leg could pass with no evidence (F4, high)api_totals_absent now fires for any pair with detail activity (compared cells or active one-sided cells) even when the bulk totals slice is empty; sol's matching-detail/empty-totals probe is a test, plus a no-false-gate test for genuinely inactive pairs (e2f0b928).
  5. CBP retrieved_at predated the retrieval (F5, medium) — the archiver captures the timestamp at the HTTP read (its signature no longer accepts an outside timestamp at all) and that manifest value feeds the facts; the round-3 build's committed evidence shows retrieved_at 23:41:40Z vs build start 23:34:50Z (1cb72f85).
  6. District facts named the wrong source member (F6, medium) — a dedicated IMDB_DISTRICT_SOURCE_LEG names the IMP_DE district-of-entry control totals in both observed_measure.source_table and source.source_table (and the extraction-method text); the margins feed keeps IMP_DETL, with the contrast asserted in tests, and the committed round-3 facts carry the corrected identity (365fcf14).
  7. Cache reuse rewrote provenance (F7, medium) — resuming over a warm cache re-minted every split marker with a fresh retrieved_at and rewrote its sidecar despite making no request. The cached overload verdict now travels with its on-disk marker and is reused verbatim — original timestamp, zero writes; the test resumes the split tree against a chmod 555 cache directory with a fetch stub that refuses to be called and proves every cache byte unchanged. The already-rewritten markers on disk were deleted and re-observed live so the committed crosscheck evidence carries only real request timestamps (365fcf14).

Review round 4 (sol, adversarial): REQUEST-CHANGES → all seven findings closed

Sol's next pass audited the closures above and returned seven edge-case refinements; each is closed with machinery and a test that re-runs sol's own probe.

  1. Fallback publication still non-atomic (high) — the exchange-less two-rename swap preserved the ENOENT window, the marker was not durable, and nothing serialized concurrent publishers. Steady-state publication is now windowless everywhere: exchange where the filesystem offers it, otherwise a symlink layout whose public name is only ever touched by one atomic rename of a prepared sibling link. First publication probes the filesystem and starts exchange-less deployments symlink-based; the only remaining window is the one-time migration of a legacy real directory on an exchange-less filesystem (plain renames cannot atomically replace a populated directory), marker-guarded with both kill points tested to roll forward. Markers are durable before every destructive step (bytes fsynced → renamed in → directory entry fsynced); an O_EXCL publisher lockfile refuses live same-host and all foreign-host holders and takes over dead same-host ones. A rename spy asserts the public path resolves before and after every rename the publisher issues (ebe3b3f6).
  2. Post-exchange strand (medium) — a hard exit between the exchange syscall and staging cleanup left the displaced previous set under .staging-* with nothing recording it. The marker is now written before the exchange, naming the staging path the syscall will park the displaced set under; a subprocess killed inside the cleanup proves recovery reclaims the orphan (ebe3b3f6).
  3. As-of parsing failed open on HTML entities (high)updated as of stripped to unmatchable text and re-created the completed-year mislabeling. _collapse_text/_clean now HTML-unescape after tag-stripping so matching runs against rendered text; the entity variant parses identically and entity-unreadable wording fails closed (d2cb2552).
  4. Impossible dates accepted (high)September 31, 2026 parsed lexically and could mark an incomplete year complete. The parser mints a real datetime.date (Sep 31 / Feb 29 in a non-leap year raise) and the fact builder compares date objects, refusing non-date endpoints including malformed retrieved_at (d2cb2552).
  5. Zero-carrier API totals counted as evidence (high) — absence was tested before the activity mask, so a totals slice of all-zero carriers passed the totals leg. Absence is now judged on active API totals gated on pair activity, and the census reconciliation leg gains the coverage half: active detail without a '-' total row is a failure, while zero-carrier detail remains agreement-by-absence (9cb22a84).
  6. Truly-empty pair false-gated (medium) — the early empty-API branch set api_totals_absent even with nothing on either side. The branch is deleted — empty slices are typed to the comparison columns and flow the one shared path; a genuinely empty pair (API 204) is asserted not to gate (9cb22a84).
  7. Write-free reuse test proved nothing (medium) — a 0555 directory still allows 0644 sidecar rewrites, and one-second timestamp precision let the pre-fix rewrite reproduce identical bytes. The test now spies every write primitive the module can reach and asserts zero write calls plus verbatim marker provenance; spy strength verified against the pre-fix ingest, where it catches the rewrite the 0555 probe missed (eaee47de).

No rebuild this round: the fixes change fail-closed paths, crosscheck gating, tests, and publication machinery — none alter ingest semantics for the archived real inputs, so the committed round-3 evidence pack remains the record (the live page and cached responses parse and aggregate identically under the new code, which is what the batteries assert).

Validation

  • Detail sums reconcile exactly (integer USD) against the publisher's own control totals on three axes (country, commodity, district) for every month — enforced in-build, zero tolerance, and the commodity axis also covers both first-unit quantity measures.
  • Consumer artifact round-trips through populace's real loader with hash verification; facts compile into ledger target specs with correct value/unit/period metadata (existing battery).
  • Vendored bridge byte-identical to upstream axiom-oracles main (sha 4a5a9698…), 241 rows, fail-closed on unmapped codes; the bulk leg's full window bridges with zero unmapped countries.

Part of #615 (P1). P2 (generator) is stacked and follows.

MaxGhenis and others added 8 commits August 4, 2026 21:53
Census International Trade API monthly HS10-by-country imports (customs
value, calculated duty, dutiable value, quantities) with byte-for-byte
response caching, SHA-256 retrieval manifests, and exact integer
reconciliation against the publisher's own totals; CBP fiscal-year
entry-summary anchors parsed from archived page bytes (exact cells only);
the vendored fail-closed Schedule C -> ISO-2 bridge (hash-pinned to the
axiom-oracles build); and consumer-artifact emission at the national,
chapter, country, and chapter-x-country grains in a populace_us_trade.*
key namespace that round-trips through load_ledger_consumer_artifact and
compiles through compile_ledger_target_references.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Census imports API times out server-side (HTTP 500 after ~150s) on
the largest chapters (vehicles, machinery, electronics), so a chapter
query that stalls past 75s or returns a compute-timeout status now fans
out into its ten 3-digit prefixes (4-digit at need). Prefixes partition
the HTS10 space, so coverage and the exact publisher-total reconciliation
are unchanged; every fetched prefix gets its own cache file and manifest
entry, the aborted attempt is recorded with superseded_by_split so
resumes skip the doomed request, and chapter-grain facts hash the full
contributing file set (set digest) when a chapter spans several files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#615 P1)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pyproject change in 723766c declared pyarrow>=15 for populace-build
but left the committed lock stale; CI regenerates on sync so it passed,
but the lock should record the resolution.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#615 P1)

Eighteen public no-auth ZIPs (census.gov/trade/downloads/{YEAR}/Merch/im_m/
IMDB{YY}{MM}.ZIP) replace thousands of 500-prone per-chapter API queries as
the margin ledger's source. The new imdb_bulk module parses the fixed-width
IMP_DETL detail per the archives' own record layouts (verified against
Documentation/*.lay in the archives; the same geometry Yale Budget Lab's
tariff model parses), validates every row's statistical period against the
archive month, and reconciles detail sums exact-integer against the
publisher's in-archive control totals by country (IMP_CTY), commodity
(IMP_COMM), and district of entry (IMP_DE).

The bulk detail upgrades two documented lane-G gaps for free: district-of-
entry margins (new feed grain + parquet, publisher's own table) and
air/vessel/containerized transport splits plus charges/CIF/second-unit
quantities and card counts (margins and detail parquet columns). The
monthly archives are year-to-date cell unions, so all-zero monthly cells
(verified on the 2025-12 archive) stay in the detail artifact but not the
margins table. Rate-provision codes carry no informal/mail/postal marker
(census.gov rpcodes.html), so no postal grounding is claimed; cty_subco and
cards_mo are carried opaquely (no published enumeration/definition).

Fact emission is source-parameterized (FactSourceLeg): record sets and fact
keys are channel-invariant, only the source provenance block varies. The
API ingest stays as the independent cross-check leg, exercised by the new
tools/crosscheck_us_import_margins_api.py over a stratified sample.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Frame checkpoints declared string columns as str(dtype), which collapses
pandas StringDtype to "str"/"string" — names pandas resolves to the
*environment-default* storage on restore. A checkpoint written with the
build's canonical python-storage strings therefore restored as
pyarrow-backed strings on any machine with pyarrow installed, the exact
serialization-boundary drift #614 canonicalized against; declaring pyarrow
for the us_trade parquet artifacts surfaced it in every venv (the wheels
gate and test_primary_qrf_frame_bank_writes_canonical_string_dtypes).

Specs now record string storage and NA marker explicitly and restores
resolve through them; the round-trip verification compares dtype objects
(str() cannot see storage). Legacy specs without the storage field restore
deterministically to python storage — what every environment those
checkpoints were verified in produced, and the build's canonical policy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconciliation now covers every measure each control file publishes (all
22 on the commodity axis, 16 on country and district) plus explicit
key-set equality in both directions — an adversarial single-digit edit to
any admitted measure, or a key present on only one side, now fails a gate
by name instead of being zero-filled away. The certification limit is
stated honestly: an all-zero YTD-carrier row contributes to no control
total (cards_mo counts current-month records) and is preserved-as-
published in the detail artifact without certification.

Corrections to overclaimed absences (all source-verified): cty_subco is
the published Country SubCode — the trade-agreement/preference-program
claim (csc.html); cards_mo is defined as "Number of Detailed Records,
Current Month"; Schedule D district 70 is "LOW-VALUED IMPORTS AND
EXPORTS", so the publisher's own low-value aggregation rides the district
margins. None of these ground postal or entry-type semantics; the module
docstring now cites the definitions instead of claiming they don't exist.

Fact provenance fails closed: months without a manifest entry refuse to
mint, empty retrieval manifests refuse to write, observed_measure carries
the actual retrieval channel, the CBP page's volatile "as of" note moves
from identity dimensions into source provenance, CBP semantic keys become
fiscal-year-invariant, district facts key on the district code alone
(names are labels, not identity), and the artifact generator pins the
Schedule C bridge hash. Adopted archives no longer claim an HTTP status
and download-manifest timestamps bind to (filename, sha256).

The cross-check tool gates on API-leg reconciliation failures and
publisher-total drift (YTD-carrier zero rows excluded honestly), reports
absent API totals, and gains an offline comparator battery. The build CLI
summarizes each month before parsing the next (December alone is 3.5M
rows) and clears stale detail partitions and reports on rerun. The
fixture's full-width row makes any one-character colspec shift fail
reconciliation, with negative tests on every control axis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…P1)

Directories under populace/build are the declarative country packages
(spec-only, manifest-carrying) unless suffixed _runtime — the governance
battery rightly rejected a new Python package at populace/build/us_trade.
The trade machinery moves to populace.build.us_runtime.us_trade, keeping
one runtime home for US build code; the vendored Schedule C bridge rides
along (wheel packaging verified at the new path).

The ACS-transfer fixtures pin their string columns to the canonical
python-storage dtype: bare list literals infer the environment-default
storage, which flips to pyarrow wherever pyarrow is installed and forced
the no-op canonicalization rewrite the identity tests assert never
happens. Note for the string-dtype owners: with pyarrow now a declared
dependency, every inferred string column is pyarrow-backed and boundary
canonicalization rewrites it to python storage — env-stable, but the
canonical-storage policy may deserve a repo-wide revisit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis and others added 20 commits August 5, 2026 10:59
…sts (#615 P1)

The two #614 raw-HDFStore assertions required the exact canonical
StringDtype (python storage) back from a plain read, but pandas
fixed-format HDF persists the logical str dtype without the storage
backend: a reading environment with pyarrow installed reconstructs
pyarrow-backed columns, so the assertions were environment-dependent and
failed once this branch's pyarrow dependency reached the full-suite job
(the 3.14 CI failures; 3.13 was fail-fast cancellation).

The production boundaries were already environment-independent — the pool
loader and the writer's round-trip verification both canonicalize after
reading (h5_io.py) — so the machinery needed no change. The tests now pin
what the file format can actually promise: the persisted logical dtype
(StringDtype with nan semantics), exact canonical restoration at the load
boundary in either environment, and exact raw reconstruction under an
explicitly pinned python string-storage option. A new regression test
holds the load boundary canonical under a forced pyarrow default,
mirroring the frame-checkpoint coverage.

Also: the changelog fragment for the P1 ingest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dpoint (#615 P1, review F4)

The archived CBP page's FY2026 cells are updated as of July 27, 2026 —
two months before the fiscal year ends — yet they were emitted as
period.type=fiscal_year completed-annual facts, so the partial entry
count (83,133,856) could be selected or compared as an annual total.

The parser now extracts the as-of note's date (as_of_date, ISO). At
emission a fiscal year is completed only when its September 30 end
predates the coverage endpoint (the publisher's note when present, else
the retrieval date, with the basis recorded). In-progress years publish
as period.type=fiscal_year_to_date with start/as_of/as_of_basis, a
fiscal_year_to_date_*_as_of_* vintage, and a separate record-set family
(…imports_revenue_collection.fytd.…): the family strip keeps the fytd
marker while dropping the year and endpoint, so a latest-eligible-period
selector over completed years can never resolve to a snapshot, and the
as-of endpoint enters the aggregate fact key so two snapshots of the
same year are distinct observations.

Also in this batch: margins-grain observed_measure now carries the
retrieval channel's own source identity instead of a hard-coded API
table name (review F5); the CBP source_file provenance names the actual
archived filename; the district grain gets an explicit
DISTRICT_ENTRY_FACT_GRAIN + ALL_IMPORT_ENTRY_FACT_GRAINS catalog; and
the consumer-artifact writer refuses retrieval entries with no
retrieved_at.

Downstream tests updated to compile the FYTD target as FYTD; new tests
cover the completed-year branch, the snapshot key disjointness, and the
retrieval-date fallback basis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…duplicate control keys (#615 P1, review F2/F3/F5/F7)

Atomic publication (F2): the build CLI now assembles everything in a
hidden staging sibling and replaces --out-dir in one directory rename.
A failed build removes its staging and leaves any previous publication
byte-for-byte untouched; a successful one leaves nothing stale beside
the new set. Two subprocess tests pin both properties (pre-populated
prior publication survives failure exactly; stale files do not survive
success).

Reconciliation evidence (F3): _reconcile_month now returns, besides the
failure list, a machine-readable record of every comparison run — per
axis key-set sizes, duplicate verdicts, and per-measure compared/matched
cell counts with both sides' integer totals. The CLI publishes it per
month under reconciliation/, plus a source_manifest.jsonl of every
retrieval, and the build report hashes both alongside the consumer
manifest's own sha256 (F5; the loader already accepts an expected
manifest hash).

Duplicate control keys (F3): a control file publishing two rows for one
key — byte-identical included — now fails the gate outright instead of
passing through set comparison and index join; value reconciliation on
that axis is skipped and recorded as such. Key shapes are enforced
(hts10 ten digits, cty_code four, dist_entry two) on detail and control
files.

Adopted-archive provenance (F5): an adopted archive with no
download-manifest match records verified_at only — retrieved_at is
deliberately absent rather than aliased to verification time, and the
consumer artifact then refuses to publish without retrieval provenance.

Cross-check gates (F7): missing API totals counterparties now gate; a
run that compared zero cells exits nonzero as an evidence failure; the
report records both parquet inputs' sha256s and every pair's fetched
API retrieval manifest. The default sample now reaches the declared
2026-06 window endpoint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…table upstream (#615 P1, review F1/F6)

Archive fidelity (F1): a committed golden pack under
tests/golden/us_trade/imdb/ carries, for IMDB2501 (2025-01) and IMDB2606
(2026-06), the official record layouts (.lay, verbatim bytes from each
archive's Documentation/), 12 verbatim raw fixed-width records per
archive selected for feature coverage (air/vessel/containerized splits,
district 70 low-value, non-default rate provisions, billion-dollar
cells), the production parse of those records as literal expected
values, and a manifest binding every file to its source archive sha256
(the same hashes as the download manifest and build report). Tests
machine-check every hard-coded production span against the official
.lay bytes — a transcription error correlated across detail and control
layouts can no longer hold — and run the committed raw lines through
the real parser against the committed literals; with the full archives
on disk the linkage test re-verifies archive hashes and re-extracts the
committed lines byte-for-byte (skips cleanly where the 100+ MB archives
are not cached).

Bridge pin (F6): BRIDGE_UPSTREAM now cites the exact axiom-oracles
commit (9310d19b8c96…) instead of blob/main; the CSV at that commit
hashes to the vendored BRIDGE_SHA256 (verified against the upstream git
object). The upstream bridge_provenance.json is vendored verbatim
beside the CSV, pinned by its own sha256, and the loader cross-binds
the two at load time: CSV hash, provenance hash, and the provenance
record's declared bridge_sha256 must all agree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ck (#615 P1, review F7)

The live 18-month run exposed a comparator false positive: after January
the Census API returns YTD-union rows (detail cells and '-' totals)
whose every month measure is zero, for pairs with activity earlier in
the statistical year; the bulk files legitimately omit them (detail) or
carry them as zero rows (controls). Zero on one channel with silence on
the other is agreement. One-sided cells and totals now gate only when
the present side carries a nonzero measure, with the dropped zero-union
counts reported per side — verified symmetric per pair on the live run
(427 = 427 totals carriers across the sample; 0 gating failures over
108,977 exact-integer cells).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ce pack (#615 P1, review F1/F3/F7)

Immutable in-branch evidence from the completed bulk run over
2025-01..2026-06: the build report (per-artifact sha256s, zero
reconciliation failures across 18 months), the source manifest (18 IMDB
archives + the archived CBP page, with URLs, sha256s, and retrieval
timestamps), per-month machine-readable reconciliation records (per-axis
key sets, duplicate-key verdicts, per-measure integer totals), and the
gated Census API crosscheck report (108,977 cells exact, 0 gating
failures, input parquets sha-bound to the build report). Golden-pack
archive sha256s equal the source-manifest rows for both golden months;
heavyweight artifacts stay outside the repo with every byte pinned here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…at the request (#615 P1, review F1/F5)

The double-rename replacement left a window in which readers of the
published path saw ENOENT and a crash stranded the previous publication
under .previous-*. Publication now exchanges staging and destination in
one syscall (renamex_np RENAME_SWAP on macOS, renameat2 RENAME_EXCHANGE
on Linux), so the published path exists at every instant; filesystems
without an exchange fall back to a two-rename swap guarded by a durable
recovery marker that the next build rolls forward (staged set intact) or
back (previous publication restored). The interruption test kills the
fallback swap between its renames and proves recovery republishes the
staged set.

The CBP page's retrieved_at was the build-start timestamp; it is now
captured at the HTTP read and the same manifest value feeds the facts —
the archiver no longer accepts an outside timestamp at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… review F2)

An unrecognized note wording parsed as 'no note', after which the fact
builder substituted the retrieval date — rewording the note to a single
fiscal year silently recreated the partial-FY bug. The as-of scan is now
scoped to the text between the entry-summaries table and the next table
(each table on the live page carries its own note; the Trade Remedy
table's differently-dated note must never become this table's coverage
endpoint) and fails closed three ways: window as-of wording the strict
pattern cannot read raises, conflicting parsed dates raise, and a window
with no note raises whenever as-of wording exists anywhere else on the
page. The single-fiscal-year wording itself now parses, so the r2 probe
yields correctly-labeled FYTD facts. The retrieval-date fallback survives
only for pages with no as-of wording at all, and the fact builder
independently refuses stats carrying a note without a parsed date.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g and gate absent totals on active pairs (#615 P1, review F3/F4)

The totals legs filtered activity on dollar measures alone, so a
zero-dollar total carrying quantity 7 was reclassified as a YTD zero
carrier and dropped; both sides now use the cell leg's exact predicate
(any nonzero dollar OR any published nonzero quantity) through one
shared mask. And the absent-API-totals flag only fired when active bulk
totals existed, letting a pair with matching dollar-active detail and
both totals slices empty pass its totals leg by comparing nothing; the
flag now fires for any pair with detail activity. Both r2 probes are
tests, plus one-sided gating for the quantity-bearing total and a
no-false-gate check for genuinely inactive pairs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s verbatim (#615 P1, review F6/F7)

District facts are minted from the archives' IMP_DE district-of-entry
control file, but both source_table members inherited the IMP_DETL
detail leg; they now carry a dedicated IMP_DE source leg (the margins
feed keeps its IMP_DETL identity).

Resuming over a warm cache re-minted every split marker with a fresh
retrieved_at and rewrote its sidecar despite making no request — reuse
is not retrieval. The cached overload verdict now travels with its
on-disk marker and is reused verbatim: original timestamp, no writes.
The test resumes the split tree against a chmod-555 cache directory
with a fetch that refuses to be called, and proves every cache byte
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, r4 build report

The F7 provenance repair completed: all sampled overload markers were
re-observed with real HTTP retrievals (stale rewrite-era sidecars
deleted first), and the gated crosscheck passed against the r4
publication — 108,977 cells across 16 pairs, 0 dollar mismatches, 0
gating failures, 0 API reconciliation failures, checked_at
2026-08-06T20:04:41Z. Chain of custody re-verified: every sha in the
committed build report matches its runtime artifact (23 checks, 0
mismatches); evidence copies byte-match the runtime originals.
Reconciliation records unchanged — the r4 rebuild was byte-identical
on all margin artifacts, so only the report, crosscheck, and source
manifest move.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…615 P1, sol review r3)

F2 (entities): the as-of scan matched raw bytes, so the live page's
pervasive &nbsp; idiom applied to the note — updated&nbsp;as of — hid
both the sentinel and the strict pattern, and an October retrieval
emitted the four in-progress FY2026 cells as completed fiscal years.
_collapse_text and _clean now HTML-unescape after tag-stripping, so
matching runs against what the page renders; the entity variants of
the r3 probe parse (keeping FYTD labels) and entity-encoded wording
the strict pattern cannot read now trips the sentinel and fails
closed instead of scanning as "no note".

F2 (dates): the note date was minted by f-string, so September 31
became "2026-09-31", which sorted after the FY2026 end and marked the
in-progress year complete. The parser now constructs a real
datetime.date (impossible dates raise), and the fact builder replaces
the lexical completeness comparison with date objects, refusing any
coverage endpoint that is not a real calendar date — including a
malformed retrieval timestamp on the fallback path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l coverage (#615 P1, sol review r3)

F4: absence was tested on the raw API totals slice, so one unrelated
all-zero carrier stood in as "totals evidence" for an active pair
while the active-row join compared nothing — one compared cell,
api_totals_absent=0, zero failures. Absence is now judged on the
active slice (zero carriers are inactivity, not evidence), and the
API's internal reconciliation gained the coverage half it was
missing: it iterated only published '-' keys, so active detail whose
total row is absent passed unreconciled; such detail now fails
("active country detail has no published '-' total row"), while
zero-carrier detail without a total stays agreement-by-absence.

F6: the r2 early empty-API branch hard-coded api_totals_absent=1,
false-gating pairs with nothing on either channel. Empty API slices
are now typed to the comparison columns and every pair flows through
the one comparison path, where a truly empty pair (204, no rows) sets
no gates — and the committed inactive-pair test now exercises exactly
that branch instead of a zero-carrier detail row (kept as its own
case).

Both r3 probes are pinned as tests; on live data every sampled pair
carries active API totals, so report numbers are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…directory (#615 P1, sol review r3)

F7: the 0555-directory probe could not prove reuse writes nothing — a
read-only directory still lets 0644 sidecars be rewritten in place,
and a rewrite landing within the marker timestamp's one-second
precision reproduces identical bytes, passing the byte-snapshot
comparison. The regression test now records every write primitive the
module can reach (Path.write_text / Path.write_bytes / writable open)
while resuming the split tree against a refusing fetch, and asserts
zero write calls plus verbatim marker provenance. Verified against
the pre-fix ingest (e2f0b92): the spy catches the sidecar rewrite
the directory probe missed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-exchange (#615 P1, sol review r3)

F1: the exchange-less fallback's two-rename swap opened an ENOENT
window on the public name. Steady-state publication is now windowless
everywhere: exchange-capable filesystems keep the single-syscall
directory exchange, and exchange-less ones use a symlink layout whose
public name is only ever touched by one atomic rename of a prepared
sibling link. The sole remaining window is the one-time migration of
a legacy real directory on an exchange-less filesystem (plain renames
cannot atomically replace a populated directory); it installs the
symlink layout so the window never recurs, and both its interruption
states roll forward under the marker.

F1b: a hard exit between the exchange syscall and its cleanup used to
strand the displaced previous set under .staging-* with nothing
recording it. The marker is now written durably (bytes fsynced, then
renamed in, then the directory entry fsynced) BEFORE every
destructive step — including the exchange — so recovery reclaims the
orphan; a publisher lockfile serializes builds toward the same
destination, with dead same-host holders taken over and live or
foreign holders refused.

Crash tests cover the retarget kill point, both migration kill
points, the post-exchange strand, in-process failures on both
layouts, and lock refusal/takeover; a rename spy proves the public
path resolves before and after every rename the publisher issues.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erive interruption state from the filesystem (#615 P1, sol review r4)

Seven publication/coverage findings from sol's round-4 pass:

- Symlink hijack: every name reaching a destructive operation — the
  public link's target, every marker-recorded directory — must be a
  plain basename in the publisher's own .<out>.<kind>-* namespace
  (joining an absolute target under the parent resolves to the target
  itself, straight into rmtree; a self-link deleted the public name).
  Foreign symlinks are refused untouched.
- Lock ABA race: the check-then-unlink stale takeover let two
  contenders each classify the same lock stale and then delete each
  other's fresh lock. The lock is now an advisory flock on a
  persistent, never-unlinked lockfile — the kernel releases it with
  its holder, so there is no staleness protocol at all; the payload
  is diagnostics for the human reading a refusal.
- Post-commit exceptions: the retarget and migration handlers derive
  the interruption point from the filesystem (does the public link
  point at the new set?) instead of control flow — an async exception
  after the commit rename used to roll the live set out from under
  the public link, and one between the migration's vacating rename
  and its flag assignment skipped restoration entirely.
- Staged durability: the staged tree is fsynced file by file before
  the first marker write; parent-directory fsyncs persist names, not
  bytes, and recovery assumes a staged set found on disk is complete.
- FY boundary: the retrieval timestamp is parsed whole (the [:10]
  prefix blessed 2026-09-30T99:99:99+00:00), a publisher as-of
  endpoint postdating the retrieval is refused, and completeness is
  strict — an endpoint ON September 30 covers a day still in
  progress, so the year stays FYTD until its end day has elapsed.
- Marker short writes: os.write loops until every byte lands.
- Recovery temp links: recovery retargets through the marker-recorded
  .linktmp-* name, so a crash mid-recovery leaves nothing unrecorded.

Committed evidence is unaffected: the archived page parses to the same
facts (FY2023-25 endpoints are strictly past their year ends; FY2026
stays FYTD) and no artifact bytes change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ity before the public name (#615 P1, sol review r5)

Five findings from sol's round-5 pass — the recovery paths lacked the
guards round 5 gave the forward paths:

- The lockfile open is O_NOFOLLOW and verifies a regular file: a
  symlink planted at the lock name no longer hands the publisher an
  arbitrary file to truncate and overwrite.
- Every recovery mode validates the public name against the states its
  own protocol can produce before mutating anything: a flip marker
  accepts only the old/new set targets (or a vacant name on first
  publication), a migration marker accepts only the marker-recorded
  set target — any other symlink used to be judged 'committed' and
  discarded the real previous publication — and the exchange and
  legacy protocols, which only ever move real directories, refuse any
  symlink. Refusals preserve the marker for the operator.
- The set name is fsynced durable before any rename touches the public
  name, in both publish paths and both recovery retargets: two renames
  in one directory carry no ordering guarantee through power loss, and
  a surviving link over a vanished set name is a dangling publication.
  The committed branches additionally require the target set to exist,
  and recovery normalizes staging into the set name first, healing the
  torn state where the link outran the set.
- _fsync_tree aborts on traversal errors instead of silently skipping
  an unreadable subtree the durability pass never saw.
- Coverage: migration recovery reuses the marker-recorded temp link
  (tested like the flip), and the timestamp probes gain the non-zero
  UTC-offset rollover and the endpoint-equals-retrieval boundary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… public and set nodes (#615 P1, sol review r6)

Seven findings from sol's round-6 pass, all corners of the round-6
guards:

- Committed recovery branches fsync the parent BEFORE deleting the
  superseded set (all four modes): recovery can run seconds after a
  live crash with the observed rename still in page cache, and a
  second power loss after the deletion persisted but the rename did
  not would leave the public name pointing at nothing with the backup
  gone.
- A symlink at the SET name refuses outright in flip and migration
  recovery: is_dir() follows symlinks, so out -> .out.set-x with
  .out.set-x -> victim used to judge 'committed' through the
  indirection and dispose the real old publication.
- Migration and legacy recovery require a real directory at the
  public name (the symlink guards missed regular files: migration
  would vacate a foreign file into previous; legacy would judge it
  committed and delete both backups). Staging/previous restores
  validate real directories too.
- Flip recovery with the public link at the set name and the set gone
  from both its set and staging names now raises with the marker
  preserved — 'publication intact' was a lie over a dangling link and
  a stranded old set.
- The UTC-rollover probe is noteless so the retrieval conversion
  itself drives classification; the ordering spy records rename
  sources too (the migration's vacate is a public mutation) and now
  also covers the recovery retarget; an unreadable staged subtree is
  proven to abort before any marker exists.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ores before the marker goes (#615 P1, sol review r7)

Five findings from sol's round-7 pass — the file-node guard applied one
level deeper, and durability on the restore path:

- The staging and set names may only hold real directories: a regular
  file at the set name refused (it would be retargeted into
  publication while the old set is deleted), and a file at the staging
  name is never normalized into the set name — flip and migration
  both.
- The previous->out restore fsyncs before the marker is removed, in
  migration and legacy: a second power loss persisting the marker
  deletion but not the restore rename left the public name absent with
  no recovery state.
- The flip 'publication intact' verdict now checks what the public
  link resolves to: a missing, regular-file, or indirect old set
  raises with the marker preserved instead of blessing a dangling or
  foreign publication.
- The ordering spies anchor on the first public-name mutation in the
  whole event stream (an unsafe early vacate can no longer escape the
  window), and the committed-recovery fsync test is parametrized over
  all four modes with the fsynced path recorded; the migration
  indirect-link probe carries its real previous backup and asserts it
  survives.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ths too (#615 P1, sol review r8)

Sol's round-8 pass confirmed every other closure and left one Medium:
the previous->out restore was fsynced on the recovery path but not on
its two siblings.

- The in-process migration failure handler fsyncs the parent right
  after restoring previous->out, before the marker removal at the end
  of the handler.
- The 'publication intact' branch fsyncs before returning: it is
  reachable on the retry AFTER an interrupted restore, blessing a
  rename that may still be sitting in page cache while the marker is
  about to be cleared.

Both paths are pinned: a live-failure spy asserts the fsync lands
between the handler's restore and the marker removal (anchored past
the failed exchange attempt's own marker withdrawal), and an
interrupted-then-retried recovery asserts the retry fsyncs before
clearing the marker.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant