Stop counting rules that measure the genre instead of the machine - #34
Merged
Conversation
Ninety texts published before generative models existed produced 888 flagged tells — a median of seven per human academic paper, with only two of the ninety coming back clean. `rhet.rule-of-three` fired on half of them. The scores were fine, because those rules weigh one or two, so the published false-positive rate was never the problem. The evidence panel was, and it costs this project more than it would cost anyone else: showing the evidence instead of a percentage is the whole argument, and a teacher who gets seven confident tells on a colleague's paper learns to disbelieve the eighth. Each rule now carries the rate at which humans use it — hits per thousand words at the ninetieth percentile of the corpus, per language, and only for rules with at least eight texts of evidence. Below that rate a finding is marked AtHumanRate: still shown, still offered to the rewriter, worth nothing to the score. Rules the corpus never sees keep counting on a single occurrence, which is why delve and tapestry are untouched. Marking rather than deleting is the second attempt. Deleting made evidence vanish as a document grew, which also handed anyone a way to bury a tell by padding; it cut the rewriter off from words it knows how to replace; and it answered "our evidence panel is noisy" by hiding evidence. All three are worse than the disease. The numbers are derived by `dotnet run --project tools/SignsOfAI.Calibration -- thresholds` and written back into the packs, so they can be regenerated when the corpus grows rather than taken on trust. The first version of this change pasted in numbers from a throwaway script while the code claimed the tool produced them, which is exactly the sort of unverifiable "measured" figure this project criticises elsewhere. Reported honestly: thresholds are fitted on the same corpus the project measures itself against, so the improvement is quoted leave-one-out — each text judged against rates derived from the other eighty-nine. That gives findings per text 9.9 -> 7.0 (-30%) and texts with nothing flagged 2 -> 9, against -32% and 10 in-sample. The recommended threshold is unchanged at 25/100 and nothing in the corpus reaches it. What this does not fix, and the code says so where it happens: rates are per rule, and language models are tuned away from repeating any one tell. The recognisable shape of machine prose is fifteen different tells at one occurrence each, every one individually at a human rate. Marking keeps that shape visible and countable; nothing here scores it. Breadth needs its own evidence, not a constant chosen today. Refs #31 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…deciding An architecture review caught this within a day of the flag landing, and it was already producing contradictory reports. The web headline counted every match while its own category chips counted only the scoring ones, so a page could say eleven signals above chips summing to three — and the share card published the inflated number. The MCP server handed an agent, under "why this reads like AI", matches the engine had already ruled out, with no field in its payload that could have revealed the difference. The CLI happened to be right because it was written last. A semantic every consumer has to remember is one some consumer will forget. AnalysisResult now partitions its own findings: Signals is the evidence and what "N signals" means everywhere, Observations is what the text uses at a rate people write at, and Findings remains everything for highlighting and for the rewriter, which should offer to replace a word whether or not its rate proves anything. MCP findings also carry RuleId and AtHumanRate now. Without them an agent had no way to tell a tell from a habit, which is precisely the judgement this project exists to hand over rather than make. A test asserts the partition and that the headline agrees with the category tallies, because that is the disagreement that actually happened. Refs #31 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 5, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the measurement half of #31. The rules that misfire on human academic writing are no longer counted as evidence — and, deliberately, are not hidden either.
What the corpus said
Ninety texts published before generative models existed produced 888 flagged tells: a median of seven per human academic paper, with only two of ninety coming back clean.
rhet.rule-of-threefired on half of them.The scores were never the problem — those rules weigh one or two, the corpus median was 6.8/100 and nothing reached the recommended threshold. The evidence panel was the problem, and it costs this project more than it would cost anyone else. Showing the evidence instead of a percentage is the entire argument, and a teacher who gets seven confident-looking tells on a colleague's paper learns to disbelieve the eighth.
What changed
Each rule now carries the rate at which people use it: hits per thousand words at the ninetieth percentile of the corpus, per language, and only for rules with at least eight texts of evidence behind them. Below that rate a finding is marked
AtHumanRate— still shown, still offered to the rewriter, worth nothing to the score. Rules the corpus never sees keep counting on a single occurrence, which is whydelveandtapestryare untouched. The mechanism is self-limiting to the genre rules by construction.Marking rather than deleting is the second attempt. Deleting was written first and it was wrong three ways: evidence vanished as a document grew (padding an essay would bury a tell), the live rewriter lost words it knows how to replace, and it answered "our evidence panel is noisy" by hiding evidence. All three are worse than the disease.
The numbers, reported the way the project reports numbers
Thresholds are fitted on the same corpus this project measures itself against, so the improvement is quoted leave-one-out — each text judged against rates derived from the other eighty-nine:
In-sample it is −32% and 10 clean texts. The smaller number is the one on the page.
rhet.rule-of-threehas left the misfire table entirely; the top entry is nowrhet.in-terms-ofat 10%.Reproducible, which was the point
derives every rate from the corpus and writes it back into the packs. Re-running it produces no diff. The first version of this change pasted numbers from a throwaway script while the code claimed the tool had produced them — an unverifiable "measured" figure, which is precisely what this project criticises in everyone else.
What it does not fix
Rates are per rule, and language models are tuned away from repeating any single tell. The recognisable shape of machine prose is fifteen different tells at one occurrence each, every one of them individually at a human rate. Marking rather than deleting keeps that shape visible and countable, but nothing here scores it. Measuring breadth instead of density needs its own evidence, not a constant chosen today.
And the limit that cannot be argued away: this removes false findings measurably, and cannot be shown to preserve true ones, because the project keeps no corpus of machine-written text — deliberately. What holds it up there is an argument, not a measurement, and it is written where it happens rather than in a commit message.
Also
HumanRatePer1000is nullable on purpose: absent means never measured, present-and-zero means a catalog turned it off. A contributed pack that rewords a rule inherits the measured rate instead of silently losing it.3 signals + 2 at a human rate).