Skip to content

Let one place decide when the tool speaks - #47

Merged
peopleworks merged 1 commit into
mainfrom
verdict-that-one-source-decides
Aug 5, 2026
Merged

Let one place decide when the tool speaks#47
peopleworks merged 1 commit into
mainfrom
verdict-that-one-source-decides

Conversation

@peopleworks

Copy link
Copy Markdown
Owner

Part one of #32. The boundary, not yet the wording.

The verdict was decided in nine places, kept in step by a comment reading "mirrors the bands in AnalysisResult.Verdict". They had already drifted: Batch.razor cut at 40 where every other surface cut at 45, so one build could colour the same document two ways.

Worse, the surfaces disagreed about whether to speak at all. EvidenceReport.VerdictHolds demanded the per-language threshold and refused to borrow the aggregate. No language in the corpus has one — English has 65 texts and Spanish 25, against the ~75 the interval needs — so the condition was false for every document in every language and the exported report never carried a verdict. Meanwhile the CLI and the web page printed a hand-picked band unconditionally, from 20, five points below the only threshold this project publishes and below the highest-scoring human text in the corpus (23.4).

One text, one engine, three answers: 90/100 "Strong signs of AI writing" on screen, and no verdict at all in the document a teacher would print and take to a committee.

The distinction that resolves it. Borrowing the aggregate error rate would misstate how often this build is wrong about Spanish — 13.3% measured against 4.1% pooled — and stays forbidden; the caveat still quotes the language's own bound. Borrowing the aggregate boundary asserts nothing about reliability. It decides when the tool speaks, it is measured and published, and it is printed beside the language's own figure so the reader weighs the right number.

Three states, because two would have regressed a test that was right:

  • a language in the corpus borrows the boundary and prints its own bound
  • a language absent from the corpus gets the score and the reason it gets nothing else, since there would be no bound on the page to correct the impression a verdict leaves
  • a fork with no calibration of its own speaks about nothing

The 45 and 70 cuts survive only to colour a reading. No human text came within twenty points of them, so the corpus can place the boundary and can say nothing whatever about those two; retiring them from the verdict's wording is part two.

Scores are untouched — bands never fed the score — and the same fixture reads 90/100 before and after, so the published calibration stands unchanged.

Adds the test that was missing: one text, three surfaces, one answer. It fails on the parent commit.

What this changes

Why

Checklist

  • dotnet test passes, with no new build warnings
  • One idea per PR

If you touched the rule packs (rules.en.json / rules.es.json):

  • The rule carries a suggestion — what the writer should do instead
  • There's a test in tests/SignsOfAI.Core.Tests covering it
  • CleanHuman in ScoringTests still scores exactly 0 (the new rule doesn't fire on ordinary prose)
  • Any regex is bounded — no unbounded .*; this runs on every keystroke in the browser
  • Spanish rules are derived from Spanish AI output, not translated from the English pack

If you touched the articles in Docs/Blog:

  • Re-ran the CLI and updated the scores the articles quote about themselves

Part one of #32. The boundary, not yet the wording.

The verdict was decided in nine places, kept in step by a comment reading "mirrors the bands in
AnalysisResult.Verdict". They had already drifted: Batch.razor cut at 40 where every other surface
cut at 45, so one build could colour the same document two ways.

Worse, the surfaces disagreed about whether to speak at all. EvidenceReport.VerdictHolds demanded
the per-language threshold and refused to borrow the aggregate. No language in the corpus has one —
English has 65 texts and Spanish 25, against the ~75 the interval needs — so the condition was false
for every document in every language and the exported report never carried a verdict. Meanwhile the
CLI and the web page printed a hand-picked band unconditionally, from 20, five points below the only
threshold this project publishes and below the highest-scoring human text in the corpus (23.4).

One text, one engine, three answers: 90/100 "Strong signs of AI writing" on screen, and no verdict
at all in the document a teacher would print and take to a committee.

The distinction that resolves it. Borrowing the aggregate *error rate* would misstate how often this
build is wrong about Spanish — 13.3% measured against 4.1% pooled — and stays forbidden; the caveat
still quotes the language's own bound. Borrowing the aggregate *boundary* asserts nothing about
reliability. It decides when the tool speaks, it is measured and published, and it is printed beside
the language's own figure so the reader weighs the right number.

Three states, because two would have regressed a test that was right:

  - a language in the corpus borrows the boundary and prints its own bound
  - a language absent from the corpus gets the score and the reason it gets nothing else, since
    there would be no bound on the page to correct the impression a verdict leaves
  - a fork with no calibration of its own speaks about nothing

The 45 and 70 cuts survive only to colour a reading. No human text came within twenty points of
them, so the corpus can place the boundary and can say nothing whatever about those two; retiring
them from the verdict's wording is part two.

Scores are untouched — bands never fed the score — and the same fixture reads 90/100 before and
after, so the published calibration stands unchanged.

Adds the test that was missing: one text, three surfaces, one answer. It fails on the parent commit.
@peopleworks
peopleworks merged commit 4ed3dc8 into main Aug 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant