Import-entry unit family P1: ledger ingest of official trade margins (#615) - #620
Open
MaxGhenis wants to merge 28 commits into
Open
Import-entry unit family P1: ledger ingest of official trade margins (#615)#620MaxGhenis wants to merge 28 commits into
MaxGhenis wants to merge 28 commits into
Conversation
Census International Trade API monthly HS10-by-country imports (customs value, calculated duty, dutiable value, quantities) with byte-for-byte response caching, SHA-256 retrieval manifests, and exact integer reconciliation against the publisher's own totals; CBP fiscal-year entry-summary anchors parsed from archived page bytes (exact cells only); the vendored fail-closed Schedule C -> ISO-2 bridge (hash-pinned to the axiom-oracles build); and consumer-artifact emission at the national, chapter, country, and chapter-x-country grains in a populace_us_trade.* key namespace that round-trips through load_ledger_consumer_artifact and compiles through compile_ledger_target_references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Census imports API times out server-side (HTTP 500 after ~150s) on the largest chapters (vehicles, machinery, electronics), so a chapter query that stalls past 75s or returns a compute-timeout status now fans out into its ten 3-digit prefixes (4-digit at need). Prefixes partition the HTS10 space, so coverage and the exact publisher-total reconciliation are unchanged; every fetched prefix gets its own cache file and manifest entry, the aborted attempt is recorded with superseded_by_split so resumes skip the doomed request, and chapter-grain facts hash the full contributing file set (set digest) when a chapter spans several files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#615 P1) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pyproject change in 723766c declared pyarrow>=15 for populace-build but left the committed lock stale; CI regenerates on sync so it passed, but the lock should record the resolution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#615 P1) Eighteen public no-auth ZIPs (census.gov/trade/downloads/{YEAR}/Merch/im_m/ IMDB{YY}{MM}.ZIP) replace thousands of 500-prone per-chapter API queries as the margin ledger's source. The new imdb_bulk module parses the fixed-width IMP_DETL detail per the archives' own record layouts (verified against Documentation/*.lay in the archives; the same geometry Yale Budget Lab's tariff model parses), validates every row's statistical period against the archive month, and reconciles detail sums exact-integer against the publisher's in-archive control totals by country (IMP_CTY), commodity (IMP_COMM), and district of entry (IMP_DE). The bulk detail upgrades two documented lane-G gaps for free: district-of- entry margins (new feed grain + parquet, publisher's own table) and air/vessel/containerized transport splits plus charges/CIF/second-unit quantities and card counts (margins and detail parquet columns). The monthly archives are year-to-date cell unions, so all-zero monthly cells (verified on the 2025-12 archive) stay in the detail artifact but not the margins table. Rate-provision codes carry no informal/mail/postal marker (census.gov rpcodes.html), so no postal grounding is claimed; cty_subco and cards_mo are carried opaquely (no published enumeration/definition). Fact emission is source-parameterized (FactSourceLeg): record sets and fact keys are channel-invariant, only the source provenance block varies. The API ingest stays as the independent cross-check leg, exercised by the new tools/crosscheck_us_import_margins_api.py over a stratified sample. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Frame checkpoints declared string columns as str(dtype), which collapses pandas StringDtype to "str"/"string" — names pandas resolves to the *environment-default* storage on restore. A checkpoint written with the build's canonical python-storage strings therefore restored as pyarrow-backed strings on any machine with pyarrow installed, the exact serialization-boundary drift #614 canonicalized against; declaring pyarrow for the us_trade parquet artifacts surfaced it in every venv (the wheels gate and test_primary_qrf_frame_bank_writes_canonical_string_dtypes). Specs now record string storage and NA marker explicitly and restores resolve through them; the round-trip verification compares dtype objects (str() cannot see storage). Legacy specs without the storage field restore deterministically to python storage — what every environment those checkpoints were verified in produced, and the build's canonical policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reconciliation now covers every measure each control file publishes (all 22 on the commodity axis, 16 on country and district) plus explicit key-set equality in both directions — an adversarial single-digit edit to any admitted measure, or a key present on only one side, now fails a gate by name instead of being zero-filled away. The certification limit is stated honestly: an all-zero YTD-carrier row contributes to no control total (cards_mo counts current-month records) and is preserved-as- published in the detail artifact without certification. Corrections to overclaimed absences (all source-verified): cty_subco is the published Country SubCode — the trade-agreement/preference-program claim (csc.html); cards_mo is defined as "Number of Detailed Records, Current Month"; Schedule D district 70 is "LOW-VALUED IMPORTS AND EXPORTS", so the publisher's own low-value aggregation rides the district margins. None of these ground postal or entry-type semantics; the module docstring now cites the definitions instead of claiming they don't exist. Fact provenance fails closed: months without a manifest entry refuse to mint, empty retrieval manifests refuse to write, observed_measure carries the actual retrieval channel, the CBP page's volatile "as of" note moves from identity dimensions into source provenance, CBP semantic keys become fiscal-year-invariant, district facts key on the district code alone (names are labels, not identity), and the artifact generator pins the Schedule C bridge hash. Adopted archives no longer claim an HTTP status and download-manifest timestamps bind to (filename, sha256). The cross-check tool gates on API-leg reconciliation failures and publisher-total drift (YTD-carrier zero rows excluded honestly), reports absent API totals, and gains an offline comparator battery. The build CLI summarizes each month before parsing the next (December alone is 3.5M rows) and clears stale detail partitions and reports on rerun. The fixture's full-width row makes any one-character colspec shift fail reconciliation, with negative tests on every control axis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…P1) Directories under populace/build are the declarative country packages (spec-only, manifest-carrying) unless suffixed _runtime — the governance battery rightly rejected a new Python package at populace/build/us_trade. The trade machinery moves to populace.build.us_runtime.us_trade, keeping one runtime home for US build code; the vendored Schedule C bridge rides along (wheel packaging verified at the new path). The ACS-transfer fixtures pin their string columns to the canonical python-storage dtype: bare list literals infer the environment-default storage, which flips to pyarrow wherever pyarrow is installed and forced the no-op canonicalization rewrite the identity tests assert never happens. Note for the string-dtype owners: with pyarrow now a declared dependency, every inferred string column is pyarrow-backed and boundary canonicalization rewrites it to python storage — env-stable, but the canonical-storage policy may deserve a repo-wide revisit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sts (#615 P1) The two #614 raw-HDFStore assertions required the exact canonical StringDtype (python storage) back from a plain read, but pandas fixed-format HDF persists the logical str dtype without the storage backend: a reading environment with pyarrow installed reconstructs pyarrow-backed columns, so the assertions were environment-dependent and failed once this branch's pyarrow dependency reached the full-suite job (the 3.14 CI failures; 3.13 was fail-fast cancellation). The production boundaries were already environment-independent — the pool loader and the writer's round-trip verification both canonicalize after reading (h5_io.py) — so the machinery needed no change. The tests now pin what the file format can actually promise: the persisted logical dtype (StringDtype with nan semantics), exact canonical restoration at the load boundary in either environment, and exact raw reconstruction under an explicitly pinned python string-storage option. A new regression test holds the load boundary canonical under a forced pyarrow default, mirroring the frame-checkpoint coverage. Also: the changelog fragment for the P1 ingest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dpoint (#615 P1, review F4) The archived CBP page's FY2026 cells are updated as of July 27, 2026 — two months before the fiscal year ends — yet they were emitted as period.type=fiscal_year completed-annual facts, so the partial entry count (83,133,856) could be selected or compared as an annual total. The parser now extracts the as-of note's date (as_of_date, ISO). At emission a fiscal year is completed only when its September 30 end predates the coverage endpoint (the publisher's note when present, else the retrieval date, with the basis recorded). In-progress years publish as period.type=fiscal_year_to_date with start/as_of/as_of_basis, a fiscal_year_to_date_*_as_of_* vintage, and a separate record-set family (…imports_revenue_collection.fytd.…): the family strip keeps the fytd marker while dropping the year and endpoint, so a latest-eligible-period selector over completed years can never resolve to a snapshot, and the as-of endpoint enters the aggregate fact key so two snapshots of the same year are distinct observations. Also in this batch: margins-grain observed_measure now carries the retrieval channel's own source identity instead of a hard-coded API table name (review F5); the CBP source_file provenance names the actual archived filename; the district grain gets an explicit DISTRICT_ENTRY_FACT_GRAIN + ALL_IMPORT_ENTRY_FACT_GRAINS catalog; and the consumer-artifact writer refuses retrieval entries with no retrieved_at. Downstream tests updated to compile the FYTD target as FYTD; new tests cover the completed-year branch, the snapshot key disjointness, and the retrieval-date fallback basis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…duplicate control keys (#615 P1, review F2/F3/F5/F7) Atomic publication (F2): the build CLI now assembles everything in a hidden staging sibling and replaces --out-dir in one directory rename. A failed build removes its staging and leaves any previous publication byte-for-byte untouched; a successful one leaves nothing stale beside the new set. Two subprocess tests pin both properties (pre-populated prior publication survives failure exactly; stale files do not survive success). Reconciliation evidence (F3): _reconcile_month now returns, besides the failure list, a machine-readable record of every comparison run — per axis key-set sizes, duplicate verdicts, and per-measure compared/matched cell counts with both sides' integer totals. The CLI publishes it per month under reconciliation/, plus a source_manifest.jsonl of every retrieval, and the build report hashes both alongside the consumer manifest's own sha256 (F5; the loader already accepts an expected manifest hash). Duplicate control keys (F3): a control file publishing two rows for one key — byte-identical included — now fails the gate outright instead of passing through set comparison and index join; value reconciliation on that axis is skipped and recorded as such. Key shapes are enforced (hts10 ten digits, cty_code four, dist_entry two) on detail and control files. Adopted-archive provenance (F5): an adopted archive with no download-manifest match records verified_at only — retrieved_at is deliberately absent rather than aliased to verification time, and the consumer artifact then refuses to publish without retrieval provenance. Cross-check gates (F7): missing API totals counterparties now gate; a run that compared zero cells exits nonzero as an evidence failure; the report records both parquet inputs' sha256s and every pair's fetched API retrieval manifest. The default sample now reaches the declared 2026-06 window endpoint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…table upstream (#615 P1, review F1/F6) Archive fidelity (F1): a committed golden pack under tests/golden/us_trade/imdb/ carries, for IMDB2501 (2025-01) and IMDB2606 (2026-06), the official record layouts (.lay, verbatim bytes from each archive's Documentation/), 12 verbatim raw fixed-width records per archive selected for feature coverage (air/vessel/containerized splits, district 70 low-value, non-default rate provisions, billion-dollar cells), the production parse of those records as literal expected values, and a manifest binding every file to its source archive sha256 (the same hashes as the download manifest and build report). Tests machine-check every hard-coded production span against the official .lay bytes — a transcription error correlated across detail and control layouts can no longer hold — and run the committed raw lines through the real parser against the committed literals; with the full archives on disk the linkage test re-verifies archive hashes and re-extracts the committed lines byte-for-byte (skips cleanly where the 100+ MB archives are not cached). Bridge pin (F6): BRIDGE_UPSTREAM now cites the exact axiom-oracles commit (9310d19b8c96…) instead of blob/main; the CSV at that commit hashes to the vendored BRIDGE_SHA256 (verified against the upstream git object). The upstream bridge_provenance.json is vendored verbatim beside the CSV, pinned by its own sha256, and the loader cross-binds the two at load time: CSV hash, provenance hash, and the provenance record's declared bridge_sha256 must all agree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ck (#615 P1, review F7) The live 18-month run exposed a comparator false positive: after January the Census API returns YTD-union rows (detail cells and '-' totals) whose every month measure is zero, for pairs with activity earlier in the statistical year; the bulk files legitimately omit them (detail) or carry them as zero rows (controls). Zero on one channel with silence on the other is agreement. One-sided cells and totals now gate only when the present side carries a nonzero measure, with the dropped zero-union counts reported per side — verified symmetric per pair on the live run (427 = 427 totals carriers across the sample; 0 gating failures over 108,977 exact-integer cells). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ce pack (#615 P1, review F1/F3/F7) Immutable in-branch evidence from the completed bulk run over 2025-01..2026-06: the build report (per-artifact sha256s, zero reconciliation failures across 18 months), the source manifest (18 IMDB archives + the archived CBP page, with URLs, sha256s, and retrieval timestamps), per-month machine-readable reconciliation records (per-axis key sets, duplicate-key verdicts, per-measure integer totals), and the gated Census API crosscheck report (108,977 cells exact, 0 gating failures, input parquets sha-bound to the build report). Golden-pack archive sha256s equal the source-manifest rows for both golden months; heavyweight artifacts stay outside the repo with every byte pinned here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…at the request (#615 P1, review F1/F5) The double-rename replacement left a window in which readers of the published path saw ENOENT and a crash stranded the previous publication under .previous-*. Publication now exchanges staging and destination in one syscall (renamex_np RENAME_SWAP on macOS, renameat2 RENAME_EXCHANGE on Linux), so the published path exists at every instant; filesystems without an exchange fall back to a two-rename swap guarded by a durable recovery marker that the next build rolls forward (staged set intact) or back (previous publication restored). The interruption test kills the fallback swap between its renames and proves recovery republishes the staged set. The CBP page's retrieved_at was the build-start timestamp; it is now captured at the HTTP read and the same manifest value feeds the facts — the archiver no longer accepts an outside timestamp at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… review F2) An unrecognized note wording parsed as 'no note', after which the fact builder substituted the retrieval date — rewording the note to a single fiscal year silently recreated the partial-FY bug. The as-of scan is now scoped to the text between the entry-summaries table and the next table (each table on the live page carries its own note; the Trade Remedy table's differently-dated note must never become this table's coverage endpoint) and fails closed three ways: window as-of wording the strict pattern cannot read raises, conflicting parsed dates raise, and a window with no note raises whenever as-of wording exists anywhere else on the page. The single-fiscal-year wording itself now parses, so the r2 probe yields correctly-labeled FYTD facts. The retrieval-date fallback survives only for pages with no as-of wording at all, and the fact builder independently refuses stats carrying a note without a parsed date. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g and gate absent totals on active pairs (#615 P1, review F3/F4) The totals legs filtered activity on dollar measures alone, so a zero-dollar total carrying quantity 7 was reclassified as a YTD zero carrier and dropped; both sides now use the cell leg's exact predicate (any nonzero dollar OR any published nonzero quantity) through one shared mask. And the absent-API-totals flag only fired when active bulk totals existed, letting a pair with matching dollar-active detail and both totals slices empty pass its totals leg by comparing nothing; the flag now fires for any pair with detail activity. Both r2 probes are tests, plus one-sided gating for the quantity-bearing total and a no-false-gate check for genuinely inactive pairs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s verbatim (#615 P1, review F6/F7) District facts are minted from the archives' IMP_DE district-of-entry control file, but both source_table members inherited the IMP_DETL detail leg; they now carry a dedicated IMP_DE source leg (the margins feed keeps its IMP_DETL identity). Resuming over a warm cache re-minted every split marker with a fresh retrieved_at and rewrote its sidecar despite making no request — reuse is not retrieval. The cached overload verdict now travels with its on-disk marker and is reused verbatim: original timestamp, no writes. The test resumes the split tree against a chmod-555 cache directory with a fetch that refuses to be called, and proves every cache byte unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, r4 build report The F7 provenance repair completed: all sampled overload markers were re-observed with real HTTP retrievals (stale rewrite-era sidecars deleted first), and the gated crosscheck passed against the r4 publication — 108,977 cells across 16 pairs, 0 dollar mismatches, 0 gating failures, 0 API reconciliation failures, checked_at 2026-08-06T20:04:41Z. Chain of custody re-verified: every sha in the committed build report matches its runtime artifact (23 checks, 0 mismatches); evidence copies byte-match the runtime originals. Reconciliation records unchanged — the r4 rebuild was byte-identical on all margin artifacts, so only the report, crosscheck, and source manifest move. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…615 P1, sol review r3) F2 (entities): the as-of scan matched raw bytes, so the live page's pervasive idiom applied to the note — updated as of — hid both the sentinel and the strict pattern, and an October retrieval emitted the four in-progress FY2026 cells as completed fiscal years. _collapse_text and _clean now HTML-unescape after tag-stripping, so matching runs against what the page renders; the entity variants of the r3 probe parse (keeping FYTD labels) and entity-encoded wording the strict pattern cannot read now trips the sentinel and fails closed instead of scanning as "no note". F2 (dates): the note date was minted by f-string, so September 31 became "2026-09-31", which sorted after the FY2026 end and marked the in-progress year complete. The parser now constructs a real datetime.date (impossible dates raise), and the fact builder replaces the lexical completeness comparison with date objects, refusing any coverage endpoint that is not a real calendar date — including a malformed retrieval timestamp on the fallback path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…l coverage (#615 P1, sol review r3) F4: absence was tested on the raw API totals slice, so one unrelated all-zero carrier stood in as "totals evidence" for an active pair while the active-row join compared nothing — one compared cell, api_totals_absent=0, zero failures. Absence is now judged on the active slice (zero carriers are inactivity, not evidence), and the API's internal reconciliation gained the coverage half it was missing: it iterated only published '-' keys, so active detail whose total row is absent passed unreconciled; such detail now fails ("active country detail has no published '-' total row"), while zero-carrier detail without a total stays agreement-by-absence. F6: the r2 early empty-API branch hard-coded api_totals_absent=1, false-gating pairs with nothing on either channel. Empty API slices are now typed to the comparison columns and every pair flows through the one comparison path, where a truly empty pair (204, no rows) sets no gates — and the committed inactive-pair test now exercises exactly that branch instead of a zero-carrier detail row (kept as its own case). Both r3 probes are pinned as tests; on live data every sampled pair carries active API totals, so report numbers are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…directory (#615 P1, sol review r3) F7: the 0555-directory probe could not prove reuse writes nothing — a read-only directory still lets 0644 sidecars be rewritten in place, and a rewrite landing within the marker timestamp's one-second precision reproduces identical bytes, passing the byte-snapshot comparison. The regression test now records every write primitive the module can reach (Path.write_text / Path.write_bytes / writable open) while resuming the split tree against a refusing fetch, and asserts zero write calls plus verbatim marker provenance. Verified against the pre-fix ingest (e2f0b92): the spy catches the sidecar rewrite the directory probe missed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-exchange (#615 P1, sol review r3) F1: the exchange-less fallback's two-rename swap opened an ENOENT window on the public name. Steady-state publication is now windowless everywhere: exchange-capable filesystems keep the single-syscall directory exchange, and exchange-less ones use a symlink layout whose public name is only ever touched by one atomic rename of a prepared sibling link. The sole remaining window is the one-time migration of a legacy real directory on an exchange-less filesystem (plain renames cannot atomically replace a populated directory); it installs the symlink layout so the window never recurs, and both its interruption states roll forward under the marker. F1b: a hard exit between the exchange syscall and its cleanup used to strand the displaced previous set under .staging-* with nothing recording it. The marker is now written durably (bytes fsynced, then renamed in, then the directory entry fsynced) BEFORE every destructive step — including the exchange — so recovery reclaims the orphan; a publisher lockfile serializes builds toward the same destination, with dead same-host holders taken over and live or foreign holders refused. Crash tests cover the retarget kill point, both migration kill points, the post-exchange strand, in-process failures on both layouts, and lock refusal/takeover; a rename spy proves the public path resolves before and after every rename the publisher issues. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erive interruption state from the filesystem (#615 P1, sol review r4) Seven publication/coverage findings from sol's round-4 pass: - Symlink hijack: every name reaching a destructive operation — the public link's target, every marker-recorded directory — must be a plain basename in the publisher's own .<out>.<kind>-* namespace (joining an absolute target under the parent resolves to the target itself, straight into rmtree; a self-link deleted the public name). Foreign symlinks are refused untouched. - Lock ABA race: the check-then-unlink stale takeover let two contenders each classify the same lock stale and then delete each other's fresh lock. The lock is now an advisory flock on a persistent, never-unlinked lockfile — the kernel releases it with its holder, so there is no staleness protocol at all; the payload is diagnostics for the human reading a refusal. - Post-commit exceptions: the retarget and migration handlers derive the interruption point from the filesystem (does the public link point at the new set?) instead of control flow — an async exception after the commit rename used to roll the live set out from under the public link, and one between the migration's vacating rename and its flag assignment skipped restoration entirely. - Staged durability: the staged tree is fsynced file by file before the first marker write; parent-directory fsyncs persist names, not bytes, and recovery assumes a staged set found on disk is complete. - FY boundary: the retrieval timestamp is parsed whole (the [:10] prefix blessed 2026-09-30T99:99:99+00:00), a publisher as-of endpoint postdating the retrieval is refused, and completeness is strict — an endpoint ON September 30 covers a day still in progress, so the year stays FYTD until its end day has elapsed. - Marker short writes: os.write loops until every byte lands. - Recovery temp links: recovery retargets through the marker-recorded .linktmp-* name, so a crash mid-recovery leaves nothing unrecorded. Committed evidence is unaffected: the archived page parses to the same facts (FY2023-25 endpoints are strictly past their year ends; FY2026 stays FYTD) and no artifact bytes change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ity before the public name (#615 P1, sol review r5) Five findings from sol's round-5 pass — the recovery paths lacked the guards round 5 gave the forward paths: - The lockfile open is O_NOFOLLOW and verifies a regular file: a symlink planted at the lock name no longer hands the publisher an arbitrary file to truncate and overwrite. - Every recovery mode validates the public name against the states its own protocol can produce before mutating anything: a flip marker accepts only the old/new set targets (or a vacant name on first publication), a migration marker accepts only the marker-recorded set target — any other symlink used to be judged 'committed' and discarded the real previous publication — and the exchange and legacy protocols, which only ever move real directories, refuse any symlink. Refusals preserve the marker for the operator. - The set name is fsynced durable before any rename touches the public name, in both publish paths and both recovery retargets: two renames in one directory carry no ordering guarantee through power loss, and a surviving link over a vanished set name is a dangling publication. The committed branches additionally require the target set to exist, and recovery normalizes staging into the set name first, healing the torn state where the link outran the set. - _fsync_tree aborts on traversal errors instead of silently skipping an unreadable subtree the durability pass never saw. - Coverage: migration recovery reuses the marker-recorded temp link (tested like the flip), and the timestamp probes gain the non-zero UTC-offset rollover and the endpoint-equals-retrieval boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… public and set nodes (#615 P1, sol review r6) Seven findings from sol's round-6 pass, all corners of the round-6 guards: - Committed recovery branches fsync the parent BEFORE deleting the superseded set (all four modes): recovery can run seconds after a live crash with the observed rename still in page cache, and a second power loss after the deletion persisted but the rename did not would leave the public name pointing at nothing with the backup gone. - A symlink at the SET name refuses outright in flip and migration recovery: is_dir() follows symlinks, so out -> .out.set-x with .out.set-x -> victim used to judge 'committed' through the indirection and dispose the real old publication. - Migration and legacy recovery require a real directory at the public name (the symlink guards missed regular files: migration would vacate a foreign file into previous; legacy would judge it committed and delete both backups). Staging/previous restores validate real directories too. - Flip recovery with the public link at the set name and the set gone from both its set and staging names now raises with the marker preserved — 'publication intact' was a lie over a dangling link and a stranded old set. - The UTC-rollover probe is noteless so the retrieval conversion itself drives classification; the ordering spy records rename sources too (the migration's vacate is a public mutation) and now also covers the recovery retarget; an unreadable staged subtree is proven to abort before any marker exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ores before the marker goes (#615 P1, sol review r7) Five findings from sol's round-7 pass — the file-node guard applied one level deeper, and durability on the restore path: - The staging and set names may only hold real directories: a regular file at the set name refused (it would be retargeted into publication while the old set is deleted), and a file at the staging name is never normalized into the set name — flip and migration both. - The previous->out restore fsyncs before the marker is removed, in migration and legacy: a second power loss persisting the marker deletion but not the restore rename left the public name absent with no recovery state. - The flip 'publication intact' verdict now checks what the public link resolves to: a missing, regular-file, or indirect old set raises with the marker preserved instead of blessing a dangling or foreign publication. - The ordering spies anchor on the first public-name mutation in the whole event stream (an unsafe early vacate can no longer escape the window), and the committed-recovery fsync test is parametrized over all four modes with the fsynced path recorded; the migration indirect-link probe carries its real previous backup and asserts it survives. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ths too (#615 P1, sol review r8) Sol's round-8 pass confirmed every other closure and left one Medium: the previous->out restore was fsynced on the recovery path but not on its two siblings. - The in-process migration failure handler fsyncs the parent right after restoring previous->out, before the marker removal at the end of the handler. - The 'publication intact' branch fsyncs before returning: it is reachable on the retry AFTER an interrupted restore, blessing a rename that may still be sitting in page cache while the marker is about to be cleared. Both paths are pinned: a live-failure spy asserts the fsync lands between the handler's restore and the marker removal (anchored past the failed exchange attempt's own marker withdrawal), and an interrupted-then-retried recovery asserts the retry fsyncs before clearing the marker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Import-entry unit family P1: ledger ingest of official trade margins
First phase of #615 — the synthetic import-entry unit family. This PR is the ledger leg: official Census/CBP series admitted as ledger-contract facts with full retrieval provenance. No entries are generated here (that is P2, stacked on this).
Source switch (2026-08-05)
P1's primary source is the Census monthly bulk imports database (IMDB), not the International Trade API. Census publishes the complete U.S. Imports of Merchandise database monthly as public no-auth ZIPs:
(URL pattern verified live 2026-08-05: all 18 archives 2025-01→2026-06 HEAD-checked 200, 118–292 MB each, ~3.3 GB total.) Eighteen GETs replace thousands of 500-prone per-chapter API queries — the API's biggest chapters (84/85/87) time out server-side after ~150 s and need recursive prefix-splitting to materialize at all. The bulk archives carry strictly more detail, and each one ships with the publisher's own control-total files and record layouts, so reconciliation runs against the archive's own published totals.
The API ingest stays in the tree as the independent cross-check leg (
census_imports.py, including the prefix-split machinery): the same series fetched over a second official channel, compared cell-for-cell on a stratified sample. Two official channels agreeing is stronger provenance than either alone. The bulk path does not depend on any API code.Reference implementation for the file geometry: Yale Budget Lab's tariff model parses these exact ZIPs (
src/io/build_import_weights.R, HS10 × country consumption imports); its six parsed field positions match the layouts shipped inside the archives (Documentation/IMP_DETL.lay), which are whatimdb_bulk.pytranscribes.What's in the box
packages/populace-build/src/populace/build/us_runtime/us_trade/:imdb_bulk.py— the primary ingest. Streams each monthly archive's fixed-widthIMP_DETLdetail (HTS-10 × Schedule C country × country subcode × district of entry × district of unlading × rate provision) with 22 monthly measures: customs value (con_val_mo), dutiable value, calculated duty, charges, CIF value, both unit quantities, card counts, and air/vessel/containerized-vessel transport splits (value/weight/charges). Every row's embedded statistical period must equal the archive month (a mislabeled row fails the ingest, never silently drops). Reconciliation is exact-integer against the publisher's own in-archive control totals on three independent axes: by country (IMP_CTY), by commodity (IMP_COMM, also the source of units of quantity), and by district of entry (IMP_DE) — any difference fails the build.census_imports.py— the API cross-check leg (unchanged from the lane-G implementation: byte-cached responses, sha-256 manifests, key elision, DET/CGP handling, publisher-totals, prefix splitting for the giant chapters).census_country_bridge.py— the Schedule C → ISO-2 bridge vendored verbatim from axiom-oraclesreference/us-tariff-panel/census_iso_bridge.csv(241 rows), hash-pinned and verified at load. Fail-closed: an unmapped country code aborts the assembly. Both legs bridge through the same table.cbp_entry_stats.py— CBP fiscal-year entry-summary anchors (total/informal entry summaries, import value, duty collected) parsed from archived bytes of the public CBP Trade Statistics page; only exact-integer cells become facts. In-progress fiscal years are encoded as fiscal-year-to-date record sets with an explicit as-of endpoint parsed from the page's own "updated as of" note (…fytd.fy2026_as_of_2026_07_27,period.type = fiscal_year_to_datewithstart/as_of); only completed fiscal years mintfiscal_yearfacts. The as-of parse fails closed (round 3): the scan is scoped to the text between the entry-summaries table and the next table — the live page carries three "updated as of" notes and the Trade Remedy table's differently-dated one must never become this table's endpoint — and any as-of wording the strict pattern cannot read, conflicting dates, or a missing table note while as-of wording exists elsewhere on the page raises instead of falling back to the retrieval date. The retrieval-date fallback survives only for pages with no as-of wording at all, and the fact builder independently refuses stats carrying a note without a parsed date. Still the only public entry-count series located; the P2 size distribution stays an explicit assumption anchored to these totals.import_entry_facts.py— consumer-artifact emission (policyengine_ledger.consumer_artifact.v1/ledger.consumer_fact.v1) at five grains now: national, chapter, country, chapter × country, and district of entry (new, from the publisher's own district table, carrying its own IMP_DE source leg — round 3: district values come from theIMP_DEcontrol file, and bothsource_tablemembers say so; the margins feed keeps itsIMP_DETLidentity). Source identity is channel-parameterized (FactSourceLeg): record-set ids and populace-namespace fact keys are identical whichever channel supplied the bytes (P3 selector stability), while each row'ssourceblock records the actual retrieval channel, file, and sha. Customs value carriesconcept_alignmentto the composed tariff spine's#input.customs_value(exact, Census definition quoted); calculated duty aligns to the total ad-valorem rate as related, never gating (Synthetic import-entry unit family: ledger-margin-generated trade entries, engine-runnable #615).Plus
tools/build_us_import_entry_margins.py(end-to-end bulk build: verify/adopt archives → parse → reconcile → staged, atomically published margins/totals/district parquets + per-month full-detail parquet + consumer artifact + build report; publication is windowless on every steady path — a single-syscall directory exchange (renamex_np(RENAME_SWAP)on macOS,renameat2(RENAME_EXCHANGE)on Linux) where the filesystem offers one, and a symlink layout elsewhere (the public name is a symlink retargeted by one atomic rename of a prepared sibling link, so it never vacates on any POSIX filesystem; round 4 replaced round 3's windowed two-rename fallback). Every destructive step is preceded by a durable fsynced recovery marker — written before the exchange too, so a set displaced by a crash mid-cleanup is reclaimed — and a publisher lockfile refuses concurrent builds toward the same destination, taking over only dead same-host holders) andtools/crosscheck_us_import_margins_api.py(the stratified, gated API comparison), and 103 trade tests (bulk ingest 35 incl. a synthetic in-test archive builder, a rename spy proving the public path resolves across every publisher rename, and subprocess kills at the retarget, inside the legacy migration, and between exchange and cleanup; archive-fidelity goldens 13, facts 26, crosscheck 17, API leg 12 — parse, reconciliation-failure, duplicate-control-key, off-period, truncation, bad-country, adoption, atomic-publication interruption/recovery, lock refusal/takeover, FYTD, as-of fail-closed mutations incl. HTML entities and impossible dates, golden-layout machine-checks, write-spied cache reuse, and offline CLI subprocess paths).What the bulk source upgrades (previously documented gaps)
census_intltrade.imports_district_entry.*), reconciled exactly against the detail.What the bulk source does and does not ground (all source-verified; corrected in review round 1)
is_postal_shipmenttherefore remains a documented P2 assumption.cty_subcois the published Country SubCode — the special trade-agreement / preference-program claim (0 = no program, A = GSP, S = USMCA, …; census.gov/foreign-trade/reference/codes/csc.html). Carried through the detail parquet as an analytic dimension; aggregated over for margins. (Round-1 correction: originally described as unenumerated.)cards_mois defined: "Number of Detailed Records, Current Month" (the published import detail file structure). It counts detail records, not entry summaries, so it is still never used as an entry-count anchor — but it now also serves as a reconciled control measure that protects every detail row with current-month activity. (Round-1 correction: originally described as undefined.)Review round 1 (sol, adversarial): REQUEST-CHANGES → hardening landed
The round-1 review found real weaknesses, all addressed in
c05884de: reconciliation expanded from 4–6 measures to every measure each control file publishes (22 commodity / 18 country / 18 district) plus two-directional key-set equality; fact provenance made fail-closed (months without manifest coverage refuse to mint, empty retrieval manifests refuse to write, channel-correctobserved_measure, bridge hash pinned in the artifact generator); CBP's volatile "as of" note moved out of identity dimensions; CBP semantic keys made fiscal-year-invariant; district facts keyed on code only; the cross-check tool now gates on API-leg reconciliation failures and publisher-total drift with an offline test battery; the build CLI streams months (bounded memory) and clears stale outputs; the fixture's full-width row makes one-character colspec shifts fail reconciliation; and three overclaimed "no published definition" statements were corrected against the sources (previous section). The checkpoint string-dtype environment-stability fix (45089b27) repairs a latent #614-class defect the new pyarrow dependency surfaced in every venv.Structure facts established empirically (recorded, tested)
*.TXT(2025:*.txt); member lookup is case-insensitive and requires exactly one match.Decisions to note
populace_us_trade.*key namespace — not an export of a PolicyEngine/ledger build — and every row'sextraction_methodplus the artifactgeneratorblock say so. The ledger repo's declarative source-package harness hand-enumerates selector rows per fact and cannot express a 10⁵–10⁶-cell series; if it grows a bulk contract, these facts relocate by reproducing the same rows there.record_set_idstayscensus_intltrade.imports_hs10.*under the source switch — the statistical series and publisher are unchanged; the channel is provenance, not identity. Semantic fact keys are therefore stable across lane G's API pulls, this build, and any future channel.Build evidence (full 18-month window, built at the hardened ingest)
facts_sha256and the manifest hash.retrieved_at, IMP_DE district source tables; current sha below). The round-3 build re-archived the live CBP page: its boilerplate bytes differ from the round-2 capture but the entry-summaries table and as-of note are byte-identical — same four exact cells, samefy2026_as_of_2026_07_27record set, no fact-identity drift.Committed evidence pack (
docs/evidence/us-import-entry-margins/)The run's claims are reviewable and recomputable in-branch, not prose:
build_report.json(per-artifact sha256s incl. all 18 per-month detail parquets and per-month reconciliation-evidence hashes),source_manifest.jsonl(18 archives + the archived CBP page: URL, sha256, size, retrieval timestamps),reconciliation/period=YYYY-MM.json× 18 (per-axis key sets, duplicate-key verdicts, per-measure compared/matched cell counts with both sides' integer totals), andcrosscheck_api_report.json(below). Chain of custody: the golden pack's archive sha256s equal the source-manifest rows for both golden months; heavyweight artifacts (3.3 GB of archives, ~1 GB of parquet) stay outside the repo with every byte pinned here.Cross-check evidence (API leg, gated — exit 0)
Stratified sample of 16 (month × chapter) pairs covering the 500-prone giants (84/85/87), small and mid chapters, both statistical years, and both window endpoints (2025-01, 2026-06): 108,977 HTS10 × country cells compared exact-integer on all four dollar measures — 0 mismatches, 0 publisher-total mismatches, 0 API-leg internal reconciliation failures, 0 absent-totals pairs. The report pins both input parquets by sha256 (equal to the build report's) and every pair's fetched API retrieval manifest. The gate fails on any mismatch, missing counterparty, or an empty comparison.
One comparator defect surfaced by the live run and fixed (
ce6753da): the API serves YTD-union rows — after January it returns cells (and-totals) with all-zero month measures for pairs active earlier in the statistical year, which the bulk detail legitimately omits. Zero on one channel with silence on the other is agreement, not divergence; these are now counted per side in the report (45,220 zero-union cells; 427 = 427 zero-carrier totals per side across the sample — the two channels carry identical carrier sets), while one-sided cells or totals carrying any nonzero measure still gate.Round 3 hardened the comparator's predicates and re-ran the gated check against the round-3 build (exit 0, same 108,977 cells, 0 everything): activity is one predicate on both legs — any nonzero dollar measure OR any published nonzero quantity — so a zero-dollar quantity-bearing total can never be misfiled as a zero carrier (F3), and an active pair with no API totals slice gates even when the bulk totals slice is empty too — the totals leg can never pass by comparing nothing (F4); both r2 probes are committed tests. The committed report also carries repaired retrieval provenance (F7): the pre-fix code had rewritten every cached split marker's
retrieved_aton each reuse, so the cached values corresponded to no actual request; the stale markers were deleted and the overloads re-observed live, meaning every timestamp in the committed report is the moment of a real HTTP event (markers from this run's live probes; split-response descendants keep their original cached retrieval timestamps).Review round 2 (sol, adversarial, on the filed PR): REQUEST-CHANGES → all findings closed
.laylayout members × both golden months verbatim plus feature-selected raw records with expected literals, sha-bound to the download manifest; a machine-check test re-derives the colspecs from the official.laybytes, and raw records run through the real parser (8c6f6e39).fcab9332).fcab9332,575a4833).cbp_entry_stats.pyabove; tests re-encoded, plus complete-year/disjointness/fallback tests) (468d0337).observed_measureis channel-correct for bulk facts; CBP source_file names the actual archived filename;verified_at/retrieved_atsplit (adopted archives without a download manifest record noretrieved_atrather than a wrong one); the artifact writer refuses manifest entries without retrieval timestamps; the build report hashes the consumer manifest (fcab9332).9310d19b…), the vendored CSV is verified byte-identical to that commit, andbridge_provenance.jsonis vendored, pinned, and cross-bound in the loader (8c6f6e39).fcab9332,ce6753da).c1bc2de5); ruff I001 ×2 fixed; changelog fragments added.Nits: measure counts corrected here to 22/18/18, window-endpoint sample now includes 2026-06, test count corrected (74 then; 91 after round 3; 103 after round 4), the district grain now rides
ALL_IMPORT_ENTRY_FACT_GRAINSbeside the four margins grains (documented: it is emitted from the publisher's district table, not the margins), and strict key-shape gates added (hts10exactly\d{10}, country/district code shapes).Review round 3 (sol, adversarial): REQUEST-CHANGES → all seven findings closed
ENOENTbetween the renames, and a kill inside the window stranded the previous publication under.previous-*. Publication now exchanges staging and destination in one syscall (renamex_np(RENAME_SWAP)/renameat2(RENAME_EXCHANGE); the test suite asserts the swap wherever the filesystem offers it, verified live on the build machine's APFS), so the published path exists at every instant. Filesystems without an exchange fall back to a two-rename swap guarded by a durable recovery marker;_recover_interrupted_publicationrolls any interruption forward (staged set intact — it was complete by construction) or back (previous publication restored) at the next build. The interruption test kills the fallback swap between its two renames withos._exit(9)in a subprocess and proves recovery republishes the staged set; in-process second-rename failure restores the previous publication immediately (1cb72f85).cbp_entry_stats.pybullet above); the single-fiscal-year wording itself now parses, so the r2 probe yields correctly-labeled FYTD facts, which is the committed mutation test. Developing this against the real archived page immediately caught the Trade Remedy table's unparseable "All programs updated as of:" note — the fail-closed design had to be scoped to the entry table's own note to be honest (130b92b9).api_totals_absent; present slice lacking the commodity → one-sided total mismatch) (e2f0b928).api_totals_absentnow fires for any pair with detail activity (compared cells or active one-sided cells) even when the bulk totals slice is empty; sol's matching-detail/empty-totals probe is a test, plus a no-false-gate test for genuinely inactive pairs (e2f0b928).retrieved_atpredated the retrieval (F5, medium) — the archiver captures the timestamp at the HTTP read (its signature no longer accepts an outside timestamp at all) and that manifest value feeds the facts; the round-3 build's committed evidence showsretrieved_at23:41:40Z vs build start 23:34:50Z (1cb72f85).IMDB_DISTRICT_SOURCE_LEGnames theIMP_DEdistrict-of-entry control totals in bothobserved_measure.source_tableandsource.source_table(and the extraction-method text); the margins feed keepsIMP_DETL, with the contrast asserted in tests, and the committed round-3 facts carry the corrected identity (365fcf14).retrieved_atand rewrote its sidecar despite making no request. The cached overload verdict now travels with its on-disk marker and is reused verbatim — original timestamp, zero writes; the test resumes the split tree against achmod 555cache directory with a fetch stub that refuses to be called and proves every cache byte unchanged. The already-rewritten markers on disk were deleted and re-observed live so the committed crosscheck evidence carries only real request timestamps (365fcf14).Review round 4 (sol, adversarial): REQUEST-CHANGES → all seven findings closed
Sol's next pass audited the closures above and returned seven edge-case refinements; each is closed with machinery and a test that re-runs sol's own probe.
ebe3b3f6)..staging-*with nothing recording it. The marker is now written before the exchange, naming the staging path the syscall will park the displaced set under; a subprocess killed inside the cleanup proves recovery reclaims the orphan (ebe3b3f6).updated as ofstripped to unmatchable text and re-created the completed-year mislabeling._collapse_text/_cleannow HTML-unescape after tag-stripping so matching runs against rendered text; the entity variant parses identically and entity-unreadable wording fails closed (d2cb2552).September 31, 2026parsed lexically and could mark an incomplete year complete. The parser mints a realdatetime.date(Sep 31 / Feb 29 in a non-leap year raise) and the fact builder compares date objects, refusing non-date endpoints including malformedretrieved_at(d2cb2552).'-'total row is a failure, while zero-carrier detail remains agreement-by-absence (9cb22a84).api_totals_absenteven with nothing on either side. The branch is deleted — empty slices are typed to the comparison columns and flow the one shared path; a genuinely empty pair (API 204) is asserted not to gate (9cb22a84).eaee47de).No rebuild this round: the fixes change fail-closed paths, crosscheck gating, tests, and publication machinery — none alter ingest semantics for the archived real inputs, so the committed round-3 evidence pack remains the record (the live page and cached responses parse and aggregate identically under the new code, which is what the batteries assert).
Validation
4a5a9698…), 241 rows, fail-closed on unmapped codes; the bulk leg's full window bridges with zero unmapped countries.Part of #615 (P1). P2 (generator) is stacked and follows.