Skip to content

CR-2026-035: fix the stale flagged claim, and what it was covering - #37

Merged
reissjason merged 1 commit into
masterfrom
cr/2026-035-benchmark-flagged-claim
Aug 26, 2026
Merged

CR-2026-035: fix the stale flagged claim, and what it was covering#37
reissjason merged 1 commit into
masterfrom
cr/2026-035-benchmark-flagged-claim

Conversation

@reissjason

Copy link
Copy Markdown
Contributor

The ask was one stale sentence. docs/SPEC-IMPLEMENTATION-STATUS.md justified a simplified C benchmark frame with "the C interpreter has no flagged or polynomial support" — CR-2026-034 gave it flagged, so half of that was false.

Fixing the sentence required checking the numbers beside it, and that turned up three larger things.

1. The numbers were not reproducible

Nothing committed produced the 20.5M ops/s in that table. src/benchmark.cpp looks like the source and isn't — it times a small interpreter defined inline in itself and never includes schema_interpreter.h. A third figure, 32M msg/s, sat three sections above it with no derivation at all.

tools/benchmark-c-interpreter.py + make bench-c regenerate both rows now:

Throughput Latency
C interpreter 8.5M ops/s 0.12 µs
Python reference 40K ops/s 25 µs
Ratio ~210x

Frame is decentlab/dl-lid.yaml — 29-byte payload, 15 fields, three plain then a two-group flagged. Three runs held 8.4–8.6M and 39–41K, so the table is rounded to ±5% and says so.

The design point worth review: the C schema is built from the same YAML the Python reference reads, by reusing the corpus harness's schema_source(). Hand-transcribing the schema into C is the obvious way to write this and the wrong one — it can drift from the YAML, and then the two rows aren't measuring the same work. The timing loop accumulates into a volatile sink, or -O2 is entitled to delete it and report an arbitrarily good number.

DL-5TM is still out of reach, but for one reason now instead of two: transform/polynomial, which C has none of.

2. Ten cells of the feature matrix were wrong about C

It credited polynomial, sqrt, abs, pow, log, floor/ceiling, clamp, all three repeat rows, and portseach of them zero occurrences in the header — and denied var, which the interpreter has had all along and which flagged reads its mask through.

The test re-derives the C column from the header rather than hardcoding my corrections, so the next capability change breaks a test instead of quietly making prose wrong again.

3. The harness's wording had misled me, in writing, twice

Its skip reasons said transform and bitfield_string were "not built by the struct API". True, but it reads as a limit on the harness. SESSION-NOTES.md and AGENTS.md both concluded from it:

The next work on C is widening the harness, not the interpreter.

Backwards. Those are the interpreter's two largest gaps — 26 schemas and 24 — and grep -cw transform include/schema_interpreter.h was 0 the whole time. The status was never "unknown because the harness cannot build them"; it was unchecked. AGENTS.md also listed ports as something C supports and the harness doesn't, which is the same error inverted — the header says outright it has no port selection.

Reasons now read the interpreter has no transform pipeline. Both documents corrected, with the lesson recorded where the mistake was made: grep the header before writing down which side a limit sits on.

Scope note

You asked for the benchmark sentence. Items 2 and 3 are here because item 1 made the document self-contradictory — the matrix credited C with polynomial three sections above my new sentence saying it has none — and because item 3 is a conclusion I published that is simply wrong. Happy to split either out if you'd rather review them separately.

Verified

  • pytest tests/ -q2783 passed / 4 skipped (was 2748; +35)
  • make test-c → 488 of 488 attempted, 0 differ
  • make selftest → ALL 3 SELFTESTS PASSED
  • make bench-c → reproduces the documented figures
  • docs/INDEX.md regenerated; no Go/Java/C# code reads anything changed here
  • bench-c deliberately not in ci — its numbers depend on the machine

🤖 Generated with Claude Code

`docs/SPEC-IMPLEMENTATION-STATUS.md` justified a simplified C benchmark frame by
saying the interpreter "has no `flagged` or `polynomial` support". CR-2026-034 gave it
`flagged`, so half the sentence was false. Fixing that sentence turned up three larger
problems behind it.

**The numbers were not reproducible.** Nothing committed produced the 20.5M ops/s in
that table. `src/benchmark.cpp` looks like the source and is not: it times a small
interpreter defined inline in itself and never includes `schema_interpreter.h`. A
third figure, 32M msg/s, sat in the same document's C entry with no derivation at all.
`tools/benchmark-c-interpreter.py` and `make bench-c` regenerate both rows now. The C
schema is built from the same YAML the Python reference reads, by reusing the corpus
harness's `schema_source()`; a hand-transcribed C copy is the obvious way to write it
and the wrong one, because it can drift from the YAML and then the two rows are no
longer measuring the same work. Measured on the frame the document names: C 8.5M
ops/s, Python 40K, ~210x, 18.2 KB stripped, 15 fields decoded on both sides. Three
runs held 8.4-8.6M and 39-41K, so the table is rounded to that precision and says so.

**Ten cells of the feature matrix were wrong about C.** It credited `polynomial`,
`sqrt`, `abs`, `pow`, `log`, `floor`/`ceiling`, `clamp`, all three `repeat` rows and
`ports` - each of them zero occurrences in the header - and denied `var`, which the
interpreter has had all along and which `flagged` reads its mask through. The test
re-derives the C column from the header rather than hardcoding the corrections, so the
next capability change breaks a test instead of quietly making prose wrong again.

**And the harness's own wording had misled me, in writing, twice.** Its skip reasons
said `transform` and `bitfield_string` were "not built by the struct API" - true, but
it reads as a limit on the harness. SESSION-NOTES.md and AGENTS.md both concluded from
it that "the next work on C is widening the harness, not the interpreter", naming
those two as harness limits whose status was "unknown because the harness cannot build
them". Backwards. They are the interpreter's two largest gaps, 26 schemas and 24, and
`grep -cw transform include/schema_interpreter.h` was 0 the whole time - the status
was never unknown, only unchecked. The reasons say "the interpreter has no transform
pipeline" now, and both documents are corrected. AGENTS.md also listed ports as
something C supports and the harness does not, which is the same error inverted.

The lesson is recorded where the mistake was made: grep the header before writing down
which side a limit sits on.

Verified: 2783 passed / 4 skipped (was 2748, +35), `make test-c` 488 of 488 with 0
differ, `make selftest` passing, docs index regenerated. `bench-c` is deliberately not
in `ci` - its numbers depend on the machine.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@reissjason
reissjason merged commit 7d6acd2 into master Aug 26, 2026
5 checks passed
@reissjason
reissjason deleted the cr/2026-035-benchmark-flagged-claim branch August 26, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant