Let one place decide when the tool speaks - #47
Merged
Conversation
Part one of #32. The boundary, not yet the wording. The verdict was decided in nine places, kept in step by a comment reading "mirrors the bands in AnalysisResult.Verdict". They had already drifted: Batch.razor cut at 40 where every other surface cut at 45, so one build could colour the same document two ways. Worse, the surfaces disagreed about whether to speak at all. EvidenceReport.VerdictHolds demanded the per-language threshold and refused to borrow the aggregate. No language in the corpus has one — English has 65 texts and Spanish 25, against the ~75 the interval needs — so the condition was false for every document in every language and the exported report never carried a verdict. Meanwhile the CLI and the web page printed a hand-picked band unconditionally, from 20, five points below the only threshold this project publishes and below the highest-scoring human text in the corpus (23.4). One text, one engine, three answers: 90/100 "Strong signs of AI writing" on screen, and no verdict at all in the document a teacher would print and take to a committee. The distinction that resolves it. Borrowing the aggregate *error rate* would misstate how often this build is wrong about Spanish — 13.3% measured against 4.1% pooled — and stays forbidden; the caveat still quotes the language's own bound. Borrowing the aggregate *boundary* asserts nothing about reliability. It decides when the tool speaks, it is measured and published, and it is printed beside the language's own figure so the reader weighs the right number. Three states, because two would have regressed a test that was right: - a language in the corpus borrows the boundary and prints its own bound - a language absent from the corpus gets the score and the reason it gets nothing else, since there would be no bound on the page to correct the impression a verdict leaves - a fork with no calibration of its own speaks about nothing The 45 and 70 cuts survive only to colour a reading. No human text came within twenty points of them, so the corpus can place the boundary and can say nothing whatever about those two; retiring them from the verdict's wording is part two. Scores are untouched — bands never fed the score — and the same fixture reads 90/100 before and after, so the published calibration stands unchanged. Adds the test that was missing: one text, three surfaces, one answer. It fails on the parent commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part one of #32. The boundary, not yet the wording.
The verdict was decided in nine places, kept in step by a comment reading "mirrors the bands in AnalysisResult.Verdict". They had already drifted: Batch.razor cut at 40 where every other surface cut at 45, so one build could colour the same document two ways.
Worse, the surfaces disagreed about whether to speak at all. EvidenceReport.VerdictHolds demanded the per-language threshold and refused to borrow the aggregate. No language in the corpus has one — English has 65 texts and Spanish 25, against the ~75 the interval needs — so the condition was false for every document in every language and the exported report never carried a verdict. Meanwhile the CLI and the web page printed a hand-picked band unconditionally, from 20, five points below the only threshold this project publishes and below the highest-scoring human text in the corpus (23.4).
One text, one engine, three answers: 90/100 "Strong signs of AI writing" on screen, and no verdict at all in the document a teacher would print and take to a committee.
The distinction that resolves it. Borrowing the aggregate error rate would misstate how often this build is wrong about Spanish — 13.3% measured against 4.1% pooled — and stays forbidden; the caveat still quotes the language's own bound. Borrowing the aggregate boundary asserts nothing about reliability. It decides when the tool speaks, it is measured and published, and it is printed beside the language's own figure so the reader weighs the right number.
Three states, because two would have regressed a test that was right:
The 45 and 70 cuts survive only to colour a reading. No human text came within twenty points of them, so the corpus can place the boundary and can say nothing whatever about those two; retiring them from the verdict's wording is part two.
Scores are untouched — bands never fed the score — and the same fixture reads 90/100 before and after, so the published calibration stands unchanged.
Adds the test that was missing: one text, three surfaces, one answer. It fails on the parent commit.
What this changes
Why
Checklist
dotnet testpasses, with no new build warningsIf you touched the rule packs (
rules.en.json/rules.es.json):suggestion— what the writer should do insteadtests/SignsOfAI.Core.Testscovering itCleanHumaninScoringTestsstill scores exactly 0 (the new rule doesn't fire on ordinary prose).*; this runs on every keystroke in the browserIf you touched the articles in
Docs/Blog: