API change for digital-convert repo + minor algo and param optim edits

Born-digital lines get the same quality answer as OCR lines (atrium-digital-convert#4, W3): the common line-quality model now scores an existing record in place. The release also carries the #3 parameter work since v1.8.0-beta. The spec stays compatible: /score_record is a new endpoint, and nothing existing changed.

POST /score_record (atrium-digital-convert#4 W3)

• In: document_json (required; any record, born-digital or OCR) and an optional pages: the record's own page keys, comma-separated (i,ii,3). A key the record lacks is 422.
• Scored: each lines[] row with text, exactly as it is: no re-reading, splitting or reordering, so every answer maps back onto its row. The scorer is the same as on every other path, batched by PPL_BATCH_LINES.
• Skipped and counted:
  • lines that carry atrium-digital-convert's decode verdict (categ Garbage or Inverted): their text layer does not decode, so the verdict stands;
  • lines without text.
• Written: lines[].categ / quality_score / lang and pages[].quality_score / quality_band, on rows that already exist. Never text, bbox or source, and never a new row.
• Out:
  • cleaned_lines: the AltoLine fields, plus the record's line. page holds the record's page key (e.g. iv), not a 1-based index;
  • pages[]: lines_scored, skipped_decode_verdict, skipped_empty, quality_score, quality_band;
  • limits_applied;
  • document_json: scored, or returned as sent when nothing was scored;
  • paradata: the run's CreateAction.
• Limits: MAX_UPLOAD_MB (the record part), ATRIUM_TEXT_INGEST_MAX_PAGES (pages scored) and ATRIUM_TEXT_INGEST_MAX_LINES_PER_PAGE (lines on one page), each 413 limit_exceeded. A record that cannot be opened, or has no doc_id, is 422 invalid_record.
• atrium-digital-convert's POST /describe calls it when OCR_POSTPROCESS_URL is set. Against an older version of this service, it falls back to /process, one page at a time. Those scores reach the /describe report only; the record is not written.

Record writes

• document_hook.write_scores() is a scoring-only write path:
  • only the scoring fields above; any other field is refused before a file is opened;
  • existing rows only;
  • lines with a decode verdict are left alone;
  • the result is validated before it is written.
• Re-vendored atrium_document.py (SCORING_FIELDS). On a record another tool originated (a born-digital one), such a merge is a scoring co-contribution: noted, not refused, and stamped contribution: "scoring". The record stays the converter's, and the fan-in accepts it. Any other write onto a foreign record is still refused unless a page asks for OCR (needs_ocr). This tool's own OCR records are unchanged.
• DIGITAL_CONVERT_KINDS gains ODT, ODS, XLSX and RTF, which atrium-digital-convert v1.1.0-beta reads with this repository's text_formats.py. This changes only the wording of the text-split report note.

Categorisation (#3)

• Witness floor (rule_short_garbage_witness_floor), new and shipped off. With [TEXT_UTILS] SHORT_GARBAGE_WITNESS_FLOOR = true, a short line the cascade rated Clear becomes Noisy when the shape witness finds garbage in it.
  • It is off in setup/config.txt and in the code, so categories do not change for any input.
  • The stage-13 figures are in the setup/config.txt comment, and docs/categorization_logic.md describes the rule. Switching it on is the maintainers' call.
• tools/ab_constant_eval.py reports gold-Clear lines sent to another class (Clear-demoted). Its discordant dump gains ref_pred / arm_pred and clear_loss rows. Developer tools only.

Note

setup/setup_api_server.sh now uses the virtualenv directory venv-ocr (it was venv-alto). An existing installation gets a fresh environment on the next run.

Assets

service_bundle.zip, openapi.json and openapi.json.sha256 (sha256sum -c openapi.json.sha256).

Full Changelog: v1.8.0-beta...v1.9.0-beta