Skip to content

Stop counting rules that measure the genre instead of the machine - #34

Merged
peopleworks merged 2 commits into
mainfrom
rules-that-misfire
Aug 4, 2026
Merged

Stop counting rules that measure the genre instead of the machine#34
peopleworks merged 2 commits into
mainfrom
rules-that-misfire

Conversation

@peopleworks

Copy link
Copy Markdown
Owner

Closes the measurement half of #31. The rules that misfire on human academic writing are no longer counted as evidence — and, deliberately, are not hidden either.

What the corpus said

Ninety texts published before generative models existed produced 888 flagged tells: a median of seven per human academic paper, with only two of ninety coming back clean. rhet.rule-of-three fired on half of them.

The scores were never the problem — those rules weigh one or two, the corpus median was 6.8/100 and nothing reached the recommended threshold. The evidence panel was the problem, and it costs this project more than it would cost anyone else. Showing the evidence instead of a percentage is the entire argument, and a teacher who gets seven confident-looking tells on a colleague's paper learns to disbelieve the eighth.

What changed

Each rule now carries the rate at which people use it: hits per thousand words at the ninetieth percentile of the corpus, per language, and only for rules with at least eight texts of evidence behind them. Below that rate a finding is marked AtHumanRate — still shown, still offered to the rewriter, worth nothing to the score. Rules the corpus never sees keep counting on a single occurrence, which is why delve and tapestry are untouched. The mechanism is self-limiting to the genre rules by construction.

Marking rather than deleting is the second attempt. Deleting was written first and it was wrong three ways: evidence vanished as a document grew (padding an essay would bury a tell), the live rewriter lost words it knows how to replace, and it answered "our evidence panel is noisy" by hiding evidence. All three are worse than the disease.

The numbers, reported the way the project reports numbers

Thresholds are fitted on the same corpus this project measures itself against, so the improvement is quoted leave-one-out — each text judged against rates derived from the other eighty-nine:

before after
findings per text 9.9 7.0 (−30%)
texts with nothing flagged 2 / 90 9 / 90
corpus median score 6.8 6.1
recommended threshold 25/100 25/100

In-sample it is −32% and 10 clean texts. The smaller number is the one on the page.

rhet.rule-of-three has left the misfire table entirely; the top entry is now rhet.in-terms-of at 10%.

Reproducible, which was the point

dotnet run --project tools/SignsOfAI.Calibration -- thresholds

derives every rate from the corpus and writes it back into the packs. Re-running it produces no diff. The first version of this change pasted numbers from a throwaway script while the code claimed the tool had produced them — an unverifiable "measured" figure, which is precisely what this project criticises in everyone else.

What it does not fix

Rates are per rule, and language models are tuned away from repeating any single tell. The recognisable shape of machine prose is fifteen different tells at one occurrence each, every one of them individually at a human rate. Marking rather than deleting keeps that shape visible and countable, but nothing here scores it. Measuring breadth instead of density needs its own evidence, not a constant chosen today.

And the limit that cannot be argued away: this removes false findings measurably, and cannot be shown to preserve true ones, because the project keeps no corpus of machine-written text — deliberately. What holds it up there is an argument, not a measurement, and it is written where it happens rather than in a commit message.

Also

  • HumanRatePer1000 is nullable on purpose: absent means never measured, present-and-zero means a catalog turned it off. A contributed pack that rewords a rule inherits the measured rate instead of silently losing it.
  • The CLI distinguishes the two in both output modes (3 signals + 2 at a human rate).
  • 9 tests for the gate; 225 core tests, 304 across the solution, all green.

peopleworks and others added 2 commits August 4, 2026 16:53
Ninety texts published before generative models existed produced 888
flagged tells — a median of seven per human academic paper, with only two
of the ninety coming back clean. `rhet.rule-of-three` fired on half of
them. The scores were fine, because those rules weigh one or two, so the
published false-positive rate was never the problem. The evidence panel
was, and it costs this project more than it would cost anyone else:
showing the evidence instead of a percentage is the whole argument, and a
teacher who gets seven confident tells on a colleague's paper learns to
disbelieve the eighth.

Each rule now carries the rate at which humans use it — hits per thousand
words at the ninetieth percentile of the corpus, per language, and only
for rules with at least eight texts of evidence. Below that rate a finding
is marked AtHumanRate: still shown, still offered to the rewriter, worth
nothing to the score. Rules the corpus never sees keep counting on a
single occurrence, which is why delve and tapestry are untouched.

Marking rather than deleting is the second attempt. Deleting made evidence
vanish as a document grew, which also handed anyone a way to bury a tell
by padding; it cut the rewriter off from words it knows how to replace;
and it answered "our evidence panel is noisy" by hiding evidence. All
three are worse than the disease.

The numbers are derived by `dotnet run --project tools/SignsOfAI.Calibration
-- thresholds` and written back into the packs, so they can be regenerated
when the corpus grows rather than taken on trust. The first version of this
change pasted in numbers from a throwaway script while the code claimed the
tool produced them, which is exactly the sort of unverifiable "measured"
figure this project criticises elsewhere.

Reported honestly: thresholds are fitted on the same corpus the project
measures itself against, so the improvement is quoted leave-one-out — each
text judged against rates derived from the other eighty-nine. That gives
findings per text 9.9 -> 7.0 (-30%) and texts with nothing flagged 2 -> 9,
against -32% and 10 in-sample. The recommended threshold is unchanged at
25/100 and nothing in the corpus reaches it.

What this does not fix, and the code says so where it happens: rates are
per rule, and language models are tuned away from repeating any one tell.
The recognisable shape of machine prose is fifteen different tells at one
occurrence each, every one individually at a human rate. Marking keeps that
shape visible and countable; nothing here scores it. Breadth needs its own
evidence, not a constant chosen today.

Refs #31

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…deciding

An architecture review caught this within a day of the flag landing, and it
was already producing contradictory reports. The web headline counted every
match while its own category chips counted only the scoring ones, so a page
could say eleven signals above chips summing to three — and the share card
published the inflated number. The MCP server handed an agent, under "why
this reads like AI", matches the engine had already ruled out, with no field
in its payload that could have revealed the difference. The CLI happened to
be right because it was written last.

A semantic every consumer has to remember is one some consumer will forget.
AnalysisResult now partitions its own findings: Signals is the evidence and
what "N signals" means everywhere, Observations is what the text uses at a
rate people write at, and Findings remains everything for highlighting and
for the rewriter, which should offer to replace a word whether or not its
rate proves anything.

MCP findings also carry RuleId and AtHumanRate now. Without them an agent
had no way to tell a tell from a habit, which is precisely the judgement
this project exists to hand over rather than make.

A test asserts the partition and that the headline agrees with the category
tallies, because that is the disagreement that actually happened.

Refs #31

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@peopleworks
peopleworks merged commit 25cf69a into main Aug 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant