v1.5.3 — the vector share is a command

72.1% here, 74.8% on Flask, 79.7% on cobra was the last published figure with no command behind it. It now comes out of benchmarks/vector_share.py, printed beside the fingerprint of the corpus each was taken on, for all three corpora in one table. Every figure this project publishes is a command.

The basis is load-bearing

The unit is bytes of the index as written: the payload re-serialized with write_index's own arguments, then again with every vector field dropped, the first checked against the file on disk before the difference is reported.

A character count reads 72.7% on this repository, whose index carries 44,913 bytes of multi-byte UTF-8 that characters discard. The two vendored corpora, almost pure ASCII, agree to within 0.001 points on either basis — so they cannot tell the two apart, and the basis has to be stated rather than inferred from a corpus that never disagrees.

Eight benchmark scripts, seven of which measure a figure the README publishes.

CI grades four of the five rulers

README section 12 names them. A, C, D and E run in .github/workflows/ci.yml, across this repository and both vendored corpora. Ruler B, the cold parse of this repository, stays a local command.

Also

• index.compact is not a setting. Compaction is the --compact flag on rag-your-code index, and benchmarks/README.md and the vector_share docstring name it that way. Five of the eight scripts print the fingerprint of the corpus they measured, which is the form docs/ROADMAP.md now states rather than quantifying over all eight.
• .ce/ and .ccm/ are ignored: a cc-enforcer index and cc-memory state are machine-local tooling, never repository content.
• The 2026-09-02 documentation audit re-derived the published figures against the code and corrected the ones that had moved.

387 tests, and src/ differs from 1.5.2 only by the version string.

Full entry: CHANGELOG.md · on PyPI: rag-your-code

v1.5.2 — one source per number, on every surface

Presentation only. No figure changed; several stopped being stated twice, and the copy PyPI renders now matches the repository.

One measurement, six numbers

The latency section put six millisecond figures in front of a reader for a single measurement: the table's, a coverage-run anecdote, 1.5.0's value, a repeat run, and two band endpoints. The table is the figure. A superseded number belongs in the changelog, not beside its replacement.

docs/ARCHITECTURE.md was a second source for that same measurement — 0.017 against 0.61 ms, "thirty-six times cheaper" — while its own sentence said the README carries the figures. It now carries the mechanism and points there.

docs/TESTING.md repeated benchmarks/README.md's account of why the displacement command exists. That account now appears once, and shorter.

One correction that is not presentational

benchmarks/README.md said two of four rulers read the live working tree. There are five.

Nothing measured moved

The corpus stamps were re-checked after the version bump and are unchanged — C 604 f84556ba7881, B 604 81e47eb0a50c, both at 0.429/0.600/0.498 and 0.314/0.471/0.381 — so every figure the README publishes still holds.

387 tests. src/ differs from 1.5.1 only by the version string.

Full entry: CHANGELOG.md · on PyPI: rag-your-code

v1.5.1 — the figure that had no command, and was wrong twice

The README is what PyPI renders, and the copy frozen into the 1.5.0 sdist still carried a figure this repository had already corrected. Going back for it found that the correction was itself wrong.

One sentence, two definitions of "test unit"

"A test declaration sometimes outranks real code" has been published since 0.6.0 and has now been wrong twice, both times because it had no committed command and each re-derivation invented its own idea of what a test is.

The second time is the instructive one. "None of cobra's 324 test units reaches rank 1" counted the 324 by file name, because Go writes command_test.go beside the code — and then asked "reaches rank 1" of a directory rule, which in a Go repository matches nothing. The claim was true only because the detector could not see one unit it was naming. One cobra test does take rank 1.

benchmarks/displacement.py is the command, with the definition in one place and language-aware: a test/tests/testdata directory anywhere above the file, or test_*.py, *_test.py, *_test.go. It separates two numbers that are not the same number — a test being top result, and a test being top result while an accepted answer sits at rank 2–3. Only the second is a cost.

ruler	questions	test units	at rank 1	displacing
C described	70	390	8	6
B cold	70	390	6	3
A Flask	35	1,126	0	0
E cobra	40	324	1	0

9 of 215 stands. What was wrong: tests reach rank 1 on 15, not 14, and Flask has 1,126 test units under one rule rather than the 1,094 that lie under tests/. What holds: none of the nine displacements is on a foreign ruler — cobra's single rank-1 test displaces nothing, its question having had no answer in the top three either way. ARCHITECTURE also said "over the three positive rulers" of a figure summed over four.

A benchmark that grades its own corpus changes it by landing

Adding that script added three units to this repository, and two of the five rulers read the live working tree. So every own-repository stamp moves: 604 units, C f84556ba7881, B 81e47eb0a50c, 317 described, 287 declined by describe.skip. The two vendored corpora are untouched — A 1,572 5fd51169eacc, E 602 3eabaa705477.

Nothing else moved with them. Re-derived on the new corpus, every score is identical to 1.5.0's: A 0.200/0.286/0.238, B 0.314/0.471/0.381, C 0.429/0.600/0.498, E 0.075/0.150/0.108, silence 0.967 own / 0.833 Flask / 0.900 cobra, and all twelve gate-ablation cells. Describing the three new units changed no question either way, checked as a set difference rather than a matching total. The vectors are still 72.1% / 74.8% / 79.7% of the three indexes.

One figure did move: the Grep head-to-head on this repository, where the baseline's top-3 goes 54.3% → 52.9% and both payloads change size (1,190,816 characters against 611,859). First place is unchanged at 22.9% against 58.6%.

The latency row is slower and the code is not

Re-measured at 604 units f84556ba7881: median 0.73 ms [0.70–0.78], p95 1.30 ms [1.17–1.39], refusal 0.016 ms, 44× cheaper. 1.5.0 published 0.49 ms at 601 units. A corpus 0.5% larger cannot account for 40%, a repeat run here read 0.69 ms, and the figure sits inside the 0.49–1.44 ms band the README already documents across releases. The machine moved, not the code.

Also

• The vector ablation was quoted over three rulers, and there are four. Re-run on all of them: ablating the default embedder costs Flask one question and cobra none, and gains the cold own-repository ruler one and the described one two at hit@3. At most two, in both directions, where ±1 had been published. The conclusion is unchanged — the vector earns close to nothing and is kept because the same storage is what an optional model needs.
• The README's gate-ablation table named no corpus. It now points at the stamps printed above it, and all twelve cells were re-measured.
• benchmarks/README.md indexes seven scripts, not six.

387 tests, unchanged. The only edit under src/ is the version string: no behaviour changed in this release, only what the project says about itself.

Full entry: CHANGELOG.md · on PyPI: rag-your-code

v1.5.0 — the constants held on a corpus nobody here chose

Closing every item on the "could be done, has not been" list. One of them — whether the skill fires unprompted — could not be closed, and is named as such rather than left off.

A third graded repository, and the first that is not Python

benchmarks/corpus/cobra/ is cobra v1.9.1 at commit 40b5bc1, Apache-2.0, 602 units of Go, graded by forty questions.

Four constants — search.min_coverage, search.min_concentration, COMMON_TERM, COVERAGE_FULL_STRENGTH — had been fitted on two corpora that were both Python and both carried their documentation inside the declaration, where the AST hands it over. Go writes it above, outside the unit's span, so this ruler grades the line scanner and the rule table on prose the parser has to pick up.

gate	E cobra hit@1/3/MRR	cobra silence
neither	0.100/0.200/0.146	0.000
coverage only	0.075/0.175/0.121	0.767
concentration only	0.075/0.150/0.108	0.833
both (shipped)	0.075/0.150/0.108	0.900

Not one constant moved. Both bars together give the best silence there, as on the other two corpora, and this one arrived four releases after the defaults were fixed. It is the closest thing to out-of-sample evidence this project has.

It also publishes the lowest number here: 0.075 hit@1, against 0.200 on Flask and 0.429 on this repository. The cause is not the language — Go documents in one terse sentence beginning with the identifier, which the parser captures correctly and which shares almost nothing with the words a user asks in. Cold retrieval tracks prose density, not language.

Chosen over two candidates by counting collisions with the absent ruler's subject words: cobra 8, gin 17, chi 18. It still fired the absence guard twice on the day it landed, and both questions were rewritten — never the corpus, and never an exemption.

The describe queue was an instruction to make retrieval worse

describe status reported 283 units pending. 270 were test functions, which 1.0.0 measured and rejected describing — five real answers lost and six false silences gained. Nothing in the tool said so, so an agent working the queue to the end would have made retrieval worse while believing the opposite.

New setting describe.skip: path patterns whose units are withheld from the queue. Empty by default, because the measurement behind it is n=1, and set in this repository's own rag-your-code.toml. What it withholds is counted and reported as declined, never subtracted in silence — a queue that shrinks without explanation reads as "nothing left to do". Patterns match with PurePosixPath.match, so tests/*.py does not reach tests/fixtures/**; describing the parser fixtures was measured to cost nothing.

describe import now reports the same remaining as describe export and bootstrap. Two disagreeing counts is how an agent decides the queue is unfinished and keeps asking for a batch that will always come back empty.

Describing a well-documented declaration loses ground

Every unit the measurement says to describe is now described — 314 of 601, with 287 withheld and counted. Twelve of the thirteen genuinely undescribed units were free or better. The thirteenth cost a graded question every way it was tried:

parser.py::_generic_units	C hit@1/3/MRR
left to its generated sentence	0.429/0.600/0.498
a long authored description	0.414/0.557/0.471
a short authored description	0.414/0.557/0.471

An authored description replaces the generated sentence, and that sentence is the only route by which the author's own docstring reaches the weight-3 description field — so writing one over a declaration whose docstring already says what a reader would search by demotes it to the weight-1 body.

Measured and rejected: appending the docstring after the authored text. It fixes that declaration and costs the corpus, 0.443 → 0.414 hit@1, because it lengthens all 314 description fields and BM25F normalises per field length. describe.skip accepts path::name so one declaration can be recorded instead.

The measured envelope was optimistic by more than noise

Re-measured on 10,000 synthetic units: full build 3.45 s (was 1.84), incremental 0.286 s (was 0.207), speedup 12.1x, isolated load 79.3 ms (was 45.4), inverted index 420 ms (was 117.7), RSS 72.2 MiB (was 58.7), and query 15.6 ms mean / 14.6 ms median — against a published 3.90 ms carried since before BM25F replaced the scoring it was taken under. Compact storage is unchanged at 35.6%. Three runs bound the spread.

Query latency on this repository is 0.49 ms median [0.45–0.58] at 601 units ac3ae43a33e7, p95 0.85 ms, refusal 0.016 ms, ~30x cheaper. The same measurement read 0.99 ms while a coverage run was in progress: idle is load-bearing, and nothing in the report can see which one it took.

Also

• Line coverage is published: 91% of 2,190 statements, with pytest --cov=ragyourcode as the command. Nothing is asserted on it — a coverage floor rewards tests that execute lines rather than tests that check.
• The four diagrams in docs/FLOW.md are parsed by a test. A mermaid block GitHub cannot render fails silently: a grey box, no error, nothing noticed. They shipped in 1.4.2 with no gate. tests/test_diagrams.py refuses an unquoted label carrying punctuation, an escaped newline where <br/> belongs, an arrow naming an undeclared node, and a style line pointing at nothing — and feeds each check a known-bad case, because a gate nobody has seen fail is a gate nobody has seen work.
• CI grades the third corpus and runs its Grep baseline.
• Two absent questions were rewritten so no subject word of that ruler occurs in any of the three graded repositories.

387 tests, up from 356. Corpus fingerprints: C 601 units ac3ae43a33e7, B 566616fbe1e7, A 1,572 5fd51169eacc, E 602 3eabaa705477.

Still not measured

Whether the skill fires unprompted in a fresh session. claude plugin eval grades exactly this and is gated behind an account-level early access this project does not have, so the suite cannot be run once to see whether it even loads. It is the only claim in this repository with no command behind it, and it is named here rather than left off the list.

Full entry: CHANGELOG.md · on PyPI: rag-your-code

v1.4.3 — three commands told an agent to run things that do not exist

A documentation audit, run as eleven parallel readers over every live document, each finding independently verified against the code before it was believed. Thirty-two survived. Three of them were in the plugin's own commands — the surface an agent reads first.

The commands told an agent to run things that do not exist

/rag-your-code:index offered --compact on bootstrap, which has no such flag: bootstrap . --compact exits 2. /rag-your-code:search sent a reader to open for results the context budget dropped, but open is an action of the JSON-lines agent protocol, not a subcommand. /rag-your-code:status said "any command warns when it is stale", and none of the three commands it prescribes does — staleness is a field (stale_index in describe status, stale in search --json), and config list never opens the index at all.

Two published numbers were wrong, not stale

The vectors are 72.1% of an index, not 65.3% — re-derived from the two indexes that ship (5,229,592 B against 1,459,238 B with the vectors removed; 74.8% on the Flask copy). The figure entered in 1.2.0 and was never re-measured through four releases.

301 units carry a written description, not 304. The sidecar has held 303 entries since 1.4.0 and two are superseded, so no state of this repository ever had 304. describe status had been answering the question all along.

A retired subject was still being quoted as current

"On the foreign ruler a test declaration is 11 of 35 top-1 results" was measured on the repository retired in 1.4.0. On Flask it is 0 of 35 — none of its 1,094 test units reaches rank 1 at all. Re-derived across the three rulers that existed then, a test displaces an accepted answer on 9 of 175 questions, where 10 had been published, and the illustrative case named in ROADMAP (test_readme_badge_match) exists in no corpus a reader can open. It was a test in the retired subject.

Also

• search.vector_recall is gated on a semantic embedder (search.py:509), so the default hash never scans a vector. README section 11 listed the full scan as an unqualified current cost while section 9 already said it stays off.
• ARCHITECTURE said "there is no network code to disable". There has been since 0.8.0; the true claim is the narrower one providers.py makes — the default provider opens no socket.
• Four of twelve settings decide what an index contains became seven of twenty-two. Two rulers became four. The golden fixture is nine units, not sixty. Flask is 1,572 units, not "eighteen hundred".
• CI now runs bootstrap, search --graph and describe promote in the clean-wheel job. It claimed to run every documented command and ran none of those three — including the one SKILL.md prescribes first.

356 tests, unchanged. Every published figure re-reproduced: C 0.443/0.614/0.509 at c9df00350cbd, B 0.314/0.471/0.383 at fb1f841fa43a, A 0.200/0.286/0.238 at 5fd51169eacc, silence 0.967 own and 0.833 foreign.

Full entry: CHANGELOG.md

v1.4.2 — the describe command was arguing from a reversed result

/rag-your-code:describe made its case with "losing 31.4% vs 34.3% on an undescribed repository". That figure came from the subject retired in 1.4.0, and it reversed when the same question was asked of a second repository: a cold index beats a Grep loop on Flask, 37.1% to 22.9%. The command now says which half depends on the repository — how much prose it already carries — and keeps the half that does not: 22.9% against 58.6% once descriptions exist.

That surface was swept for fingerprints and unit counts in both 1.3.0 and 1.4.0 and passed each time, because neither sweep looked for bare percentages. The sweep was the thing that was wrong.

docs/FLOW.md draws the whole thing in four diagrams: the loop a user lives in, indexing, the answering path with both evidence bars and all four refusal reasons, and the three surfaces — four commands, one skill, and the nine-action JSON-lines protocol.

The README now says how to update the plugin, not only how to install it. claude plugin update rag-your-code is refused with Plugin "rag-your-code" not found, which reads like the plugin is gone rather than like the argument is short: it wants the full name@marketplace id, and the marketplace cache has to be refreshed first.

356 tests, unchanged — no .py file was touched, so every corpus fingerprint published in 1.4.1 still holds.

v1.4.1 — stop printing the docstring twice

Found by installing 1.4.0 from PyPI into a clean environment and using it on an unfamiliar repository — the one thing no test in this project does.

A generated description ends with the author's own docstring, because that is how a docstring becomes searchable. The block then printed the source below it, which contains the same docstring. On Flask, 2,381 of 3,382 characters of prose header were a verbatim repeat of the code beneath it — a fifth of everything a query returned, paid for twice.

It never showed on this project's own rulers: the descriptions here are agent-written and are not in the source, so there was nothing to duplicate. It took a corpus whose authors wrote docstrings, which arrived one release ago.

_visible_description drops the quoted half when the code below already shows it, and keeps it when it does not — an authored description is the one part of a block a reader cannot recover by reading the code. searchable_text is untouched, so the docstring stays exactly as findable and no ruler moved.

The saving is in answers, not characters. The budget was already the binding constraint, so the same 12,000 characters now carry more distinct code:

declarations delivered at one budget	before	after
Flask, 30 answered	92	119
this repository, 60 answered	305	323

Also: "Documented intent:" was a literal in three modules that have to agree on it — annotate writes it, parser writes it again for the other fourteen languages, and document looked for it to decide which declarations are undocumented. It now has one home.

Republished with fresh fingerprints: B 584 fb1f841fa43a, C 584 c9df00350cbd. Ruler A is untouched at 5fd51169eacc — the vendored corpus does not move when this repository does, which is the point of vendoring it.

356 tests (+2, 0 removed, by node-id set diff against v1.4.0). Full detail in CHANGELOG.md.

v1.4.0 — the ruler's subject now lives here

The foreign ruler graded a repository on one machine. It is now a copy of Flask 3.1.3 carried in this repository — and repeating the measurements against a corpus that holds still reversed two conclusions this project had been publishing.

benchmarks/corpus/flask/ is Flask 3.1.3 at commit 22d9247, BSD-3-Clause, docs/ and .git/ omitted and the rest as published: 1,572 units of code nobody here wrote, indexed with no descriptions. rag-your-code.toml keeps it out of this project's own index — without that, eighteen hundred units of a web framework enter two rulers that measure retrieval over this repository and falsify a third asserting nothing here answers thirty questions. Neither the wheel nor the sdist carries it.

The local model is worse or identical on every ruler. 1.1.0 read the comparison as "better on every positive ruler". Its largest gain was on ruler A, whose two arms had been taken against two states of a repository being edited while the script ran. Repeated with one fingerprint per row: A 0.200/0.286/0.238 against MiniLM's 0.171/0.257/0.214, B identical, C 0.443/0.614/0.507 against 0.429/0.600/0.500. The B and C gains were one or two questions — inside the noise of a two-unit change to the corpus, and they reverse sign under it. The extra stays shipped for the cross-language and paraphrase pairs the hash scores zero on, which these rulers cannot see.

Concentration does not subsume coverage. On the retired subject the two bars were interchangeable. Here they are not: together they silence 0.833 of the foreign absent questions, against 0.800 for concentration alone and 0.733 for coverage alone. One question, and the first measurable reason in four releases to keep both.

Which side wins against Grep depends on the repository. The README said flatly that a Grep loop wins on an undescribed one; that was a single subject generalised. On Flask it reverses — 37.1% right file first against 22.9%, no descriptions added — because Flask has docstrings to retrieve against. What does not depend on the subject is what descriptions buy: 58.6% against 22.9% here.

Foreign silence is 0.833, down from 0.933, and the cause is a limit rather than a defect: a word counts as evidence unless it reaches 5% of units, so how, when and are are ubiquitous in a repository with 304 written descriptions and discriminating across 1,572 short undocumented methods. A sweep says the shipped 0.28 stays — every step up buys that silence out of the answers.

CI now runs the foreign ruler, the foreign absent ruler and the foreign head-to-head, none of which was possible before. On a different OS it reproduced 5fd51169eacc / 0.200 / 0.286 / 0.238 / 0.833 exactly.

354 tests (+1, 0 removed, by node-id set diff against v1.3.0). Full detail in CHANGELOG.md.

v1.3.0 — every published figure is a command

1.2.1 declared the roadmap empty. Checking that claim the way this project checks everything else — by running the commands — turned up four published figures with no command behind them, and one ruler that had stopped running.

Two harnesses that should always have existed. Query latency and the entire Grep head-to-head in README section 7 — the strongest claim the project makes — were published from scripts that were never committed. benchmarks/grep_baseline.py now defines the baseline in code: drop the words the corpus shows are everywhere, one substring search per remaining word, rank by distinct words hit, ties on path. It reproduced the ranked side exactly (58.6% / 75.7% / 60 of 70) and moved the baseline's, which is the expected shape. Both directions still hold: Grep wins on an undescribed repository 40.0% to 28.6%, and loses 22.9% to 58.6% on a described one.

Latency, to the precision it has. benchmarks/query_latency.py repeats the whole measurement and prints the spread. Ten runs across one hour spanned 0.62–1.44 ms median and 1.09–7.34 ms p95 — wider than any change the code has ever made to this number. The README now says 0.65 ms and 1.13 ms with the range beside them. The old 0.83 / 1.68 sits inside that range: unfalsifiable rather than wrong.

Ruler A had stopped running. Two questions pointed at a declaration the foreign repository had since moved and renamed. Its integrity check refused to grade rather than scoring them as misses — exactly what that check is for, and it had been failing unnoticed because nothing ran it. Repointed and re-measured on a corpus grown from 1,267 to 1,345 units: 0.229 / 0.371 / 0.286, where 0.400 and 0.300 were the corpus, not retrieval.

Every table now carries its corpus. The README has said since 1.1.0 that every report carries a fingerprint. Every report did; none of the tables printing them did. They do now — A 471b78f9f806, B dca2a4656659, C 0a2f050dbafd. The visible cost of not having done this earlier: the local-model comparison has an A row whose two arms were taken against two states of a repository being edited while the script ran. It stays, with the confound named.

Counts nobody had re-derived. "all 32 questions" in a file that has held 30 since its only commit; "126 questions" matching no set here; "silences 47%" that measures 0.600. All four now name the command or the table instead.

353 tests (+1, 0 removed, by node-id set diff against v1.2.1). Full detail in CHANGELOG.md.

v1.2.1 — the claim about what cannot be checked was itself unchecked

A documentation correction, and the claim it corrects had shipped in three releases.

Since 1.0.0 docs/ROADMAP.md has carried a section titled "the one thing a command cannot check", asserting that whether the bundled skill fires unprompted "needs an interactive session with a model deciding, which no test in this repository can stand in for".

That was wrong. claude plugin eval runs eval cases against a plugin, and its own --ablation help names the mechanism exactly:

graders marked with-only, incl. tool_used: Skill, are a plugin-fired indicator rather than part of the score

— with a no-plugin baseline arm for contrast. The property is measurable, and the roadmap had been asserting the opposite without ever checking it against the tooling it was describing.

Corrected status: measurable, and not measured

Two real blockers, now stated in place of the false one:

• claude plugin eval reports plugin eval is currently in early access on this account, so neither it nor its init scaffold will run. The gate is a server-side entitlement — there is no eval key in the local experiment cache and no local flag.
• The case schema is not publicly documented. The plugins reference covers skills, agents, hooks, MCP, LSP, monitors, themes, output styles, workflows, channels and dependencies — and not eval suites. Three fragments of --help text are the only description available.

Why no suite was written anyway

A case.yaml assembled from guessed field names could not be executed even once to see whether it loads. A suite that silently fails to load reads as a gate while checking nothing — which is precisely the defect this project shipped twice with an install line naming a package index it did not publish to, and once with a threshold sweep in which every setting scored identically. Guessing the schema would have reproduced the exact class of failure the roadmap exists to record.

What would close it

Written down instead of performed: early access granted, claude plugin eval init for the real schema, cases split between prompts that should fire the skill and matched controls that should not, --ablation with-without, and --runs above the default because a stochastic trigger cannot be read off one sample.

Since 1.2.0 the property also matters less than it did — four commands give a deterministic entry path that does not depend on a model choosing to fire anything.

No code changed. 352 tests unchanged, CI 11/11.