A bitemporal warehouse for official statistics, and a build that fails when the publisher rewrites history.
What is in it
- 703 kB of real Office for National Statistics data, four series, 535 published versions walked and 23,943 recorded changes, committed rather than fetched so every measurement here runs offline.
- A dbt gate that fails when a load changes a value for a period the warehouse already holds, unless that revision is declared in a ledger. Watched failing with the ledger emptied (7 undeclared release groups) and watched passing with it, on every CI run.
- The same as-of definition written three ways and run against ClickHouse, DuckDB and PostgreSQL, compared on every period rather than on the headline one.
- Three loads into an Apache Iceberg table on a real S3 endpoint, producing 5 snapshots, one of which records a load that published nothing.
The measurement worth reading
A ReplacingMergeTree keyed on (series, period, vintage) with vintage as a Date looks like it keeps every version and does not. Over the whole corpus it keeps 23,872 of 23,943 rows: 71 published values deleted, with no error anywhere. A release date is not a unique key, because a statistical office can issue two versions on the same morning, and this corpus contains mornings where it did. That key was this repository's own recommendation until the measurement was run over the whole corpus instead of one example, and the wrong design is kept in the evidence beside the right one.
Verification
8 required status checks on main, four of which re-run a measurement and fail if a single byte of the result changed, transcripts included. The demo needs nothing but Python.