Skip to content

Phase 0 conformance harness, and close the query-concurrency gap - #64

Merged
mo4islona merged 4 commits into
masterfrom
spec-suite
Jul 26, 2026
Merged

Phase 0 conformance harness, and close the query-concurrency gap#64
mo4islona merged 4 commits into
masterfrom
spec-suite

Conversation

@mo4islona

Copy link
Copy Markdown
Contributor

Builds Phase 0 of the conformance program from spec/13's build order — the harness stubs, the structural validators, deterministic seeding, and one end-to-end smoke — then uses it to close the first P0 gap in the register.

Follows #63, which landed the spec suite itself.

Phase 0 harness

tests/harness/ implements HC-1/2/3/5/8/12:

HC-1 scheduler.rs Network-state JSON + gzipped FlatBuffers assignment over HTTP, built with the real sqd-assignments builder (crypto-box headers included)
HC-2 origin.rs Chunk files over HTTP with a byte ledger + injectors (delay, stall, status, corrupt, truncate)
HC-3 portal.rs Seeded keys, genuinely signed queries, per-field deviation knobs
HC-8 registry.rs Programmable sqd_contract_client::Client — epoch advance, allocation change, read failure
HC-5 validators.rs Query-response, log-page and status validators
HC-12 seed.rs + corpus.rs Labelled RNG streams off one root seed, printed on panic; chunks generated, not checked in

Test binaries are one per subject: e2e (assign → download → query → verify → logs pull), query_surface (RP-10/RP-20/RP-1 outcomes), query_concurrency (separate by necessity — the OB signals are process-global, so a gauge assertion can't share a process with other query-running tests).

The crate gains a library target: an integration tier can't reach a binary-only crate, and MG-4 wants the conformance subset timed as its own CI job rather than buried in the unit-test run.

What it does not cover

The input side (IB-40/41/42/43/44) is genuinely black-box over HTTP, but the query side enters below the transport at p2p::{validate_query, execute, build_delivery, build_log}. So IB-1/2 and the P2PController intake — queue capacity, reject fan-out, the assignment pending queue — are not under test. That's recorded in harness::UNCOVERED and in a new "SUT boundary, as built" note in spec/13, and declared_gaps_cite_the_spec enforces that every such declaration names a real spec identifier.

GAP-1 closed

run_query bound its scopeguard to _, so the guard dropped at the end of its own let statement rather than at the end of the function. The slot was released before the query ran: max_parallel_queries was unenforced (the real ceiling was the transport's message-handler product) and the running-query gauge fell back to zero as fast as it rose, so overload was invisible in metrics as well as unbounded in fact.

Test landed first and was checked both ways — fails on the defect, passes on the fix, and stable over 10 consecutive runs. The register row is gone; RP-4, REQ-22, INV-31 and PF-1 no longer carry the exception, and P-Q-PAR reads observed-enforced in spec/15.

Two incidental fixes

  • src/cli.rssentry_is_enabled is declared positional (like its env-only siblings) but a bare bool defaults to a flag action, which clap rejects for positionals. Every debug-build invocation aborted, --help included. Release compiled the assert out, which is why it survived.
  • .gitattributes — the repo never had one. LFS was configured and the index held correct pointers, but nothing routed *.parquet through the filter, so git compared 356 MB of smudged content against 131-byte pointers and called tests/data permanently modified — one git add -A from inlining all of it. Stored oids match the working tree exactly, so declaring the filter reconciles the tree without touching the data.

Also included

The freshness-attribution spec work: a rejection whose reference input is the worker's own clock is a worker fault, not a client one (ADR-20, INV-26, FM-55, GAP-33).

Verification

47 tests across 4 binaries, zero ignored. cargo test --all-features, cargo fmt --check, clippy at the CI deny level, and spec/tools/check_spec.py all clean.

Next

GAP-2 (externally supplied content can terminate the process) and GAP-3 (no reconciliation deletion floor) remain open at P0. Their HC-1 fault knobs — UnparseableFileUrl, NoChunksForWorker — exist, but no tests drive them yet.

🤖 Generated with Claude Code

mo4islona and others added 2 commits July 25, 2026 14:26
Every unchecked row in spec/13's traceability matrix was blocked on the same
thing: no way to feed the worker an assignment, serve it chunks, and ask it a
signed question. This adds the stubs (HC-1/2/3/8), the structural validators
(HC-5) and deterministic seeding (HC-12), plus the smoke path that proves they
compose. The crate grew a library target because an integration tier cannot
reach a binary-only crate, and MG-4 wants the conformance subset timed as its
own gate rather than buried in the unit-test run.

The SUT boundary is a compromise worth naming: the input side (IB-40/41/42/43/44)
really is black-box over HTTP, but the query side enters below the transport at
p2p::{validate_query, execute, build_delivery, build_log}, so the controller's
intake and IB-1/2 stay untested. That is recorded in harness::UNCOVERED and in
spec/13 rather than papered over, and a test keeps those declarations pointing
at real identifiers.

Two findings fall out. GAP-1 is confirmed: its test fails on the inert cap and
passes on the one-token fix, so it ships quarantined for Phase 1. And the CLI
could not be parsed at all in a debug build — a bare bool positional trips a
clap assertion, aborting even --help — which release builds hid by compiling
the assertion out.

Also carries the freshness-attribution spec work: a rejection whose reference
input is the worker's own clock is a worker fault, not a client one (ADR-20,
INV-26, FM-55, GAP-33).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`run_query` bound its scopeguard to `_`, so the guard dropped at the end of its
own `let` statement rather than at the end of the function. The slot was
released before the query ran, which left `max_parallel_queries` unenforced —
the real ceiling was the transport's message-handler product — and made the
running-query gauge fall back to zero as fast as it rose, so overload was
invisible in metrics as well as unbounded in fact.

The test lands first and was checked both ways: it fails on the defect and
passes on the fix. It lives in its own binary because the OB signals are
process-global `lazy_static`s, and a gauge assertion sharing a process with
other query-running tests would be reading their traffic — passing for the
wrong reason, and blind to a regression.

Closes the register's GAP-1: RP-4, REQ-22, INV-31 and PF-1 no longer carry the
exception, and P-Q-PAR is observed-enforced.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both test jobs died with SIGTERM part-way through compiling the crate — no
rustc error, only "the runner has received a shutdown signal". The library
target and three integration binaries took `cargo test` from two heavy link
targets to six, and four concurrent codegen units over the polars/arrow
monomorphisations exhaust a 16 GB runner. Clippy passing over the same
--all-targets set is the tell: it emits metadata and never links. So CI drops
DWARF nothing reads, caps codegen on the two jobs that link, and stops the unit
job rebuilding the conformance binaries the conformance job already owns.

The harness had copied the status report, the log-page budget and the overload
retry hint out of the controller rather than calling them. The budget copy was
already 100 KiB looser than production, so the RP-22 page bound was checked
against a number the worker never uses. A harness that asserts against a
reimplementation of its subject can pass while the subject is broken, so those
three are now reachable from p2p and called.

test_fs asserted a raw directory listing, leaving it red on any macOS checkout
Finder had touched. Every production caller of `ls` already filters by name
shape, so the filtering belongs in the test and not in `ls`.

`conformance_metrics` read as "metrics about conformance" rather than
"conformance tests that touch process-global metrics", and `conformance.rs` was
a catch-all that would have silted up as the CT-2/4/9 classes land. Each file
is now named for what it tests, so the next class has an obvious home instead
of a default one.

The split of query_concurrency stays, but for the reason that actually forces
it: one test binary is one process, and the OB signals are process-global, so a
gauge assertion cannot sit beside tests that run queries for other reasons.
That is now stated where someone deciding where to put a new test will read it.

The harness self-test moves into the harness module — it guards the harness's
own honesty, so it should hold wherever the harness is used rather than in
whichever binary happened to be first.

Also adds the .gitattributes the repo never had. LFS was configured and the
index held correct pointers, but nothing routed *.parquet through the filter,
so git compared 356 MB of smudged content against 131-byte pointers and called
tests/data permanently modified — one `git add -A` away from inlining all of it
into the repo. The stored oids match the working tree exactly, so declaring the
filter reconciles the tree with no change to the data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mo4islona
mo4islona force-pushed the spec-suite branch 2 times, most recently from 7868574 to 753d370 Compare July 26, 2026 11:14
`fetch_add` then compare let every concurrent rejection carry the count, and the
OB-6 gauge with it, above max_parallel_queries until its guard ran. The cap on
executing queries still held, but the gauge lied (INV-31) and the inflated count
shed queries that would have fit. `fetch_update` takes a slot only when one is
free, and the gauge moves only once admission has succeeded.

The harness named this worker on every chunk in the assignment document, and the
reader resolves membership from the chunk's own worker_indexes rather than from
the worker's chunk list. So `assigned: false` and NoChunksForWorker both built a
document that assigned everything — GAP-3's advertised fault input was inert, and
would have stayed inert under a test written against it.

declared_gaps_cite_the_spec checked identifier shape, which NOTREAL-999 satisfies,
and took a token's leading digits, which resolved a mistyped GAP-3x to GAP-3. It
now requires the digits to reach the end of the token and the id to exist in spec/,
parsed by the three definition forms check_spec.py recognises. Continuations like
the 2 of IB-1/2 carry no prefix and are still unchecked.

The cap assertion was `peak <= cap`, which a worker stuck at a single slot also
satisfies; equality catches that, and running at two caps is what distinguishes
honouring the configured value from honouring some ceiling. The test keeps
join_all, because one task running every admission check in a burst is what makes
the rejection certain — a task per query lets early ones finish first and goes
flaky. What it needed was a sampler that can see a few instructions, so that
thread spins rather than sleeping.

The tier only ran under --all-features, so mvcc-chunks was always on and its 37
cfg sites never took the other branch, while the Dockerfile builds default
features. It now runs both, matrixed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@mo4islona
mo4islona merged commit 5c98f0d into master Jul 26, 2026
6 checks passed
@mo4islona
mo4islona deleted the spec-suite branch July 26, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant