Minor refinements + sidecar from David

What's new

• LOWPPL_CLEAR_MAX 50 → 35 (#3). A line of three or more words now takes the language-model shortcut to Clear only below perplexity 35. From 35 to 50 it goes through the score routing like any other line.
  • Over both collections it moves 62,859 lines (0.09 %) in 16,687 documents: Noisy → Clear 37,062, Clear → Noisy 12,342, Noisy → Trash 9,272, Clear → Trash 3,900, Trash → Noisy 279, Trash → Clear 4. The band changes on 1,128 of 1,226,349 pages.
  • On the labelled sets it fixes 63 lines and breaks none: 60 of @david-spacil's and 3 of the 2,067 gold (510 → 507 errors). @DanaKriv's lines do not move.
  • This closes the pilot's parameter work. The witness floor and the shape witness stay off.
• No band for a page without text lines (batch path). aggregate_STAT.py gave a page made only of Empty / Non-text lines the band Clear in its document record. It now omits quality_band there, as the service already did. In the July collections that was 48,544 pages (4 %). The page-stats CSV has no band column and is unchanged.
• tools/corpus_change.py (#3, stages 16–18):
  • reach re-scores whole collections in memory under several configurations and writes only the changed lines, with labels joined and page aggregates (--pages, --page-ref). pagecheck applies the stage-17 rule to its summary.
  • recat re-scores a collection into a new folder, sharded and resumable (--expect-missing).
  • diff compares two re-scored trees, lines and pages, with an independent recount (--verify). It pairs lines by (page_num, line_num), so a document stored in another order is counted, not refused.
• tools/ab_constant_eval.py (both copies): the exact McNemar test no longer overflows when more than 1,023 lines disagree.
• Labels (tools/gold/, see GOLD.md):
  • sidecars/floor_2026-10-05_keys.csv: @david-spacil's floor annotation, keys and labels only, on 141,292 lines of the July batch;
  • one gold label corrected: sektlll (CTX201703691 149:287) is Noisy, not Trash.
• Docs: README.md on what avg_quality_score leaves out, and docs/categorization_logic.md on the new threshold. The pre-commit size limit is raised to 6,024 KB for the sidecar.

Compatibility

• Categories change for lines of three or more words with a perplexity from 35 to 50, and, through the page rules, for a few lines next to them. Re-scoring a stored batch with this version moves about 0.09 % of its lines.
• In document records from the batch path, pages without text lines no longer carry quality_band. Nothing else in records, the API or openapi.json changes.

The ARUP and ARUB collections are re-scored with this version as DOC_LINE_CATEG_1210, with page stats (#3, stage 18). They are delivered separately, not as release assets.

Full Changelog: v1.10.0-beta...v1.11.0-beta