A one-line fix to a metric, and a great deal of measurement that changed what we believe about our own
system. Nothing in the data path moved.
The fix
nuthatch_sealed_through reported 0 after every restart while the query path knew the true
watermark all along. The gauge was only ever written when a seal happened, so between a restart and the
next seal /metrics said 0 and /sql provenance said the truth. Two surfaces disagreeing about one
fact, and the wrong one is where Prometheus looks.
It has always done this - a 2.7.1 nest restarted alongside a 3.0.0 one behaves identically - but it
matters more now that 3.0.0 ships six nuthatch_entity_* series inviting alerts on that surface. An
alert of the form "sealed_through has not advanced" fired after every restart of a perfectly healthy
nest, and an alert that cries wolf gets muted.
It now seeds from the store at startup, where the watermark has always been durable.
What we measured, and what it changed
None of this alters the binary. It is recorded because a claim without a number is an opinion, and
several of ours turned out to be wrong.
Sealed segments are never compacted. The oldest production nest holds 10,923 files at a 6 KB
median, and the smallest is one row in 545 bytes. It costs: a table with 3.1x more rows spread over
3.1x fewer segments answers COUNT(*) 20% faster. A file costs roughly 0.14-0.18 ms.
docs/bench/segment-layout.md.
The benchmark noise floor now has a number, and it has two halves. For a single client, medians of
15 runs reproduce to 3% - but a single run varies by up to 44%, and 92% on the cheapest query. Under
concurrency the median stops being the right statistic entirely: at four clients it falls to 12 ms
while p95 holds at 92 ms, because the distribution goes bimodal and the median tracks the fast mode
while hiding the tail. Throughput saturates around 120 req/s. docs/bench/noise-floor.md.
The Linux binary has two ABI floors and the README named one. It links libstdc++ - because it
embeds DuckDB - and needs GLIBCXX_3.4.29. Every platform we list clears it, so this was
incompleteness rather than a broken promise, but a reader on new glibc with an old libstdc++ met a
requirement nobody had stated.
We do not verify that the chain we indexed is the chain consensus agreed on. Hash-linking proves
the headers we were served form an internally consistent chain. It does not prove that chain is the
canonical one, because settling that needs consensus-layer data nuthatch does not read. An endpoint
that lies consistently is indistinguishable, to us, from one that tells the truth.
docs/verification.md now says so as a stated non-claim.
RFC-0042, for those following it
Slice 0 is complete on both release targets. DuckDB is 93% of native artefact bytes and the sole
reason a C++ runtime is linked - but it is 10.6% of clean build time on Linux and 8.0% on macOS,
while wasmtime and cranelift are roughly twice that on both. The RFC's premise that it dominates
build time is measured false.
Six DuckDB roles, not the four §9 assumed, and two are product-visible: the admissible function
vocabulary for entities.toml comes from duckdb_functions(), and the engine version string is
written into grafting identity.
The RFC is now unfrozen in full. Nothing has been decided about removing anything.