import-run: a published archive into a local Postgres, one command - #22
Merged
Conversation
The single biggest onboarding hole (2026-08 review): DATA.md publishes every census run as a free tar.zst archive, but nothing could put one INTO a database — so a contributor could not run `cargo run -p api` against real data without sweeping a live chain themselves. `import-run` is export-run's inverse. It accepts an archive (extraction shells out to `tar --zstd`, falling back to `zstd -dc | tar`; no new crate dependency) or an already-extracted directory, reads either manifest shape (sweep-time RunManifest or a RebuiltManifest with `rebuilt_at` and any exporter provenance fields a newer branch adds — every shape-varying field is optional, unknown fields are ignored), and writes runs, agent_snapshots, check_results, and http_archive in one transaction, batched via UNNEST. Choices worth stating: - token_id stays TEXT end to end and is bound `::numeric` — a uint256 does not fit i64, same rule the sweeper and export-run follow. - A run that already exists is refused unless --replace is passed, which deletes that run's rows first, in the same transaction — an interrupted import leaves the old run or the new one, never a mix. - http_archive gets summary columns only (bodies are never in archives), no row at all for an agent whose export carries no fetch summary, and `scheme` — NOT NULL but absent from the export — is recovered from rung 2's own evidence, URI prefix as fallback. - A chain the local database has not seen is seeded as a minimal DISABLED row from the manifest's chain_id/registry; the run's status is 'finished' only when the manifest says finished_at, else 'unknown' — the vocabulary migration 0014 defines for "cannot say". - Raw NULs (a hazard tokenURI() has actually delivered) are escaped with the store's existing convention before TEXT, and recursively across evidence trees before jsonb. Verified end to end against Postgres 16: the celo archive (7833fc49-a5b7-477b-99ce-946f650f0064) imports as 9,747 snapshots, 68,229 check results (9,747 × 7 rungs), 9,518 archive summaries, and /api/runs then serves the run. README gains "Local data without sweeping"; DATA.md gains the matching paragraph and states the one asymmetry (no bodies) where the archives' contents are described. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 4, item 18 of the post-review work order — the review's biggest contributor unlock: DATA.md publishes every run free, but nothing could load one, so running the API locally meant sweeping live chains.
crates/sweeper/src/bin/import-run.rs:DATABASE_URL=… cargo run -p sweeper --bin import-run -- [--replace] <archive.tar.zst | dir>. Idempotent (refuses an existing run without--replace; replace is one transaction), zero new dependencies (shells totar --zstd, documented), parses both manifest shapes incl. the exporter fields from Provenance: the index records the checker that swept, not the exporter #15,token_idstays a string into NUMERIC, missing chains auto-seed disabled from the manifest./api/runs+/ratesserved correctly,--replaceand extracted-dir paths exercised, 8 new unit tests, fmt/clippy clean.🤖 Generated with Claude Code