CR-2026-035: fix the stale flagged claim, and what it was covering - #37
Merged
Merged
Conversation
`docs/SPEC-IMPLEMENTATION-STATUS.md` justified a simplified C benchmark frame by saying the interpreter "has no `flagged` or `polynomial` support". CR-2026-034 gave it `flagged`, so half the sentence was false. Fixing that sentence turned up three larger problems behind it. **The numbers were not reproducible.** Nothing committed produced the 20.5M ops/s in that table. `src/benchmark.cpp` looks like the source and is not: it times a small interpreter defined inline in itself and never includes `schema_interpreter.h`. A third figure, 32M msg/s, sat in the same document's C entry with no derivation at all. `tools/benchmark-c-interpreter.py` and `make bench-c` regenerate both rows now. The C schema is built from the same YAML the Python reference reads, by reusing the corpus harness's `schema_source()`; a hand-transcribed C copy is the obvious way to write it and the wrong one, because it can drift from the YAML and then the two rows are no longer measuring the same work. Measured on the frame the document names: C 8.5M ops/s, Python 40K, ~210x, 18.2 KB stripped, 15 fields decoded on both sides. Three runs held 8.4-8.6M and 39-41K, so the table is rounded to that precision and says so. **Ten cells of the feature matrix were wrong about C.** It credited `polynomial`, `sqrt`, `abs`, `pow`, `log`, `floor`/`ceiling`, `clamp`, all three `repeat` rows and `ports` - each of them zero occurrences in the header - and denied `var`, which the interpreter has had all along and which `flagged` reads its mask through. The test re-derives the C column from the header rather than hardcoding the corrections, so the next capability change breaks a test instead of quietly making prose wrong again. **And the harness's own wording had misled me, in writing, twice.** Its skip reasons said `transform` and `bitfield_string` were "not built by the struct API" - true, but it reads as a limit on the harness. SESSION-NOTES.md and AGENTS.md both concluded from it that "the next work on C is widening the harness, not the interpreter", naming those two as harness limits whose status was "unknown because the harness cannot build them". Backwards. They are the interpreter's two largest gaps, 26 schemas and 24, and `grep -cw transform include/schema_interpreter.h` was 0 the whole time - the status was never unknown, only unchecked. The reasons say "the interpreter has no transform pipeline" now, and both documents are corrected. AGENTS.md also listed ports as something C supports and the harness does not, which is the same error inverted. The lesson is recorded where the mistake was made: grep the header before writing down which side a limit sits on. Verified: 2783 passed / 4 skipped (was 2748, +35), `make test-c` 488 of 488 with 0 differ, `make selftest` passing, docs index regenerated. `bench-c` is deliberately not in `ci` - its numbers depend on the machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ask was one stale sentence.
docs/SPEC-IMPLEMENTATION-STATUS.mdjustified a simplified C benchmark frame with "the C interpreter has noflaggedorpolynomialsupport" — CR-2026-034 gave itflagged, so half of that was false.Fixing the sentence required checking the numbers beside it, and that turned up three larger things.
1. The numbers were not reproducible
Nothing committed produced the 20.5M ops/s in that table.
src/benchmark.cpplooks like the source and isn't — it times a small interpreter defined inline in itself and never includesschema_interpreter.h. A third figure, 32M msg/s, sat three sections above it with no derivation at all.tools/benchmark-c-interpreter.py+make bench-cregenerate both rows now:Frame is
decentlab/dl-lid.yaml— 29-byte payload, 15 fields, three plain then a two-groupflagged. Three runs held 8.4–8.6M and 39–41K, so the table is rounded to ±5% and says so.The design point worth review: the C schema is built from the same YAML the Python reference reads, by reusing the corpus harness's
schema_source(). Hand-transcribing the schema into C is the obvious way to write this and the wrong one — it can drift from the YAML, and then the two rows aren't measuring the same work. The timing loop accumulates into avolatilesink, or-O2is entitled to delete it and report an arbitrarily good number.DL-5TM is still out of reach, but for one reason now instead of two:
transform/polynomial, which C has none of.2. Ten cells of the feature matrix were wrong about C
It credited
polynomial,sqrt,abs,pow,log,floor/ceiling,clamp, all threerepeatrows, andports— each of them zero occurrences in the header — and deniedvar, which the interpreter has had all along and whichflaggedreads its mask through.The test re-derives the C column from the header rather than hardcoding my corrections, so the next capability change breaks a test instead of quietly making prose wrong again.
3. The harness's wording had misled me, in writing, twice
Its skip reasons said
transformandbitfield_stringwere "not built by the struct API". True, but it reads as a limit on the harness.SESSION-NOTES.mdandAGENTS.mdboth concluded from it:Backwards. Those are the interpreter's two largest gaps — 26 schemas and 24 — and
grep -cw transform include/schema_interpreter.hwas0the whole time. The status was never "unknown because the harness cannot build them"; it was unchecked.AGENTS.mdalso listed ports as something C supports and the harness doesn't, which is the same error inverted — the header says outright it has no port selection.Reasons now read
the interpreter has no transform pipeline. Both documents corrected, with the lesson recorded where the mistake was made: grep the header before writing down which side a limit sits on.Scope note
You asked for the benchmark sentence. Items 2 and 3 are here because item 1 made the document self-contradictory — the matrix credited C with
polynomialthree sections above my new sentence saying it has none — and because item 3 is a conclusion I published that is simply wrong. Happy to split either out if you'd rather review them separately.Verified
pytest tests/ -q→ 2783 passed / 4 skipped (was 2748; +35)make test-c→ 488 of 488 attempted, 0 differmake selftest→ ALL 3 SELFTESTS PASSEDmake bench-c→ reproduces the documented figuresdocs/INDEX.mdregenerated; no Go/Java/C# code reads anything changed herebench-cdeliberately not inci— its numbers depend on the machine🤖 Generated with Claude Code