GHA fixes + shared scripts update

Patch release on top of v1.15.0-beta, and the S2 pilot candidate (atrium-project#77).

Fixed

• A document_json / document_record whose pages or lines is not a list of objects is refused with 422 invalid_record on /process and /score_record; it used to end in a 500 (atrium-project#72). The batch reports the same error instead of an AttributeError.

Changed

• The invalid_record description in openapi.json names this case; no endpoint, field or status changes.
• CI: the hub's workflows are pinned to the commit with the shared fix, including the Trivy database retry.

Images this release certifies (images.json, atrium-project#81):

• ghcr.io/ufal/atrium-ocr-postprocess:1.15.1-beta = sha256:e1ed70c817400d6d2509db7a5f09532a7e246b2385ac931c17b62909b9a6665c
• ghcr.io/ufal/atrium-ocr-postprocess-api:1.15.1-beta = sha256:6928575c68679fcff4c76b13f04003ccae7b4f847aeb529a747c6695280e2a5c

Full Changelog: v1.15.0-beta...v1.15.1-beta

API-CLI alignment of results

v1.15.0-beta: the batch and the service agree at the edges

What changed (atrium-project#72)

• The batch's LayoutReader method keeps an ALTO page without WIDTH/HEIGHT and a word without coordinates, as the service does. Before, it dropped them.
• A page that runs out of GPU memory even at the smallest LayoutReader window keeps the ALTO's own order, as in the service. Before, the batch skipped the page.
• The service reads its perplexity model from [CLASSIFY] MODEL_NAME in setup/config.txt, as the batch does; GPT2_MODEL_NAME still comes first. The shipped config names the same model as before (Qwen2.5-0.5B).
• The batch record's page quality_score is rounded to four decimals, as the service's.
• Both web frontends compute "Avg Quality" from Clear and Noisy lines only, as the API does.
• /info lists every line field (original_lang, orig_lang_score, page, page_label added); the spec's field descriptions are corrected.

Compatibility

• The API is unchanged apart from the version. Batch outputs now include lines and pages that were dropped before.

Upgrade notes

• Pin 1.15.0-beta. Changing [CLASSIFY] MODEL_NAME now changes the service's model too, unless GPT2_MODEL_NAME is set.

Images this release certifies (images.json, atrium-project#81):

• ghcr.io/ufal/atrium-ocr-postprocess:1.15.0-beta = sha256:7c4fb7eceda555a4e6d996e40ad93fd2250061c68fdfa542853c3925d029b88c
• ghcr.io/ufal/atrium-ocr-postprocess-api:1.15.0-beta = sha256:8090aace5db566ca6b24e3fa15a85912fe2423422869632c6d0eb0c0a5588ebc

Full Changelog: v1.14.0-beta...v1.15.0-beta

API-CLI parity fixes

v1.14.0-beta: the same lines and page score from the command line and the service

What changed (atrium-project#72)

• Both sides now keep blank lines as Empty lines, numbered with the rest, so the service's line_num matches the batch's for the same page.
  • In ALTO, blank lines are the paragraph breaks that the text extraction writes.
  • In plain text and documents, blank lines are kept only with [TEXT_INGEST] KEEP_BLANK_LINES = true. The service now reads this setting too. The default stays false.
• reading_order=alto now reads the page with alto-tools, as --method alto-tools does: a blank line before each text block, and a hyphenated line joined to the next. alto_tools.py (Apache-2.0) is now part of the service image and bundle.
• A page's quality_score from /process, /score_record and score_record.py is now the mean of its Clear and Noisy lines, as in the batch page statistics. It used to include Trash lines.
• docs/usage.md explains what counts as a blank line.

Compatibility

• The API fields are unchanged. The spec differs only in its version and two descriptions.
• An ALTO upload's cleaned_lines now includes Empty lines, so the lines after them get higher line_num values. A record written by the service gets those rows too.
• Plain-text and document uploads give the same lines as before unless KEEP_BLANK_LINES is set.
• A page's quality_score changes if the page has Trash lines. A page with only Trash lines now has no score.
• With reading_order=alto, an ALTO file in a namespace that alto-tools does not know now gets a 422.

Upgrade notes

• Pin 1.14.0-beta.
• A client that expects only lines with text should skip lines whose category is Empty.

Images this release certifies (images.json, atrium-project#81):

• ghcr.io/ufal/atrium-ocr-postprocess:1.14.0-beta = sha256:a2f00af5d09b439c9ef9b031297e46ed540ef9c11ac4426ac5226ee395b32d8a
• ghcr.io/ufal/atrium-ocr-postprocess-api:1.14.0-beta = sha256:880b025c57ada6194dcd109ae4a2ad6b75b013f9adf7d79a7c1dca495154a4ce

Full Changelog: v1.13.1-beta...v1.14.0-beta

API-CLI alignment and citation updated

v1.13.1-beta: the release v1.13.0-beta did not get

What changed

• The v1.13.0-beta tag pushed its images, but no GitHub release was made: the service bundle was missing record_scoring.py. It ships now, and tests/test_release_bundle.py runs the release's bundle check on every push.
• .github/production-image.json no longer lists score_record.py (it is the command line, which the service does not reach), so Workflow Policy Lint passes again.
• Everything from v1.13.0-beta is included:
  • text with form feeds and multi-page JSON come back page by page from /process;
  • the opt-in reading_order=alto and include_record;
  • score_record.py on the command line;
  • one split-word merge shared by the command line and the service.

Compatibility

• The API is unchanged apart from the version.

Upgrade notes

• Pin 1.13.1-beta. The 1.13.0-beta images exist but have no release notes.
  Images this release certifies (images.json, atrium-project#81):

• ghcr.io/ufal/atrium-ocr-postprocess:1.13.1-beta = sha256:c0ac7d12c3bca2f3461b71e4cebf603d53afa909e4768b47df8cdcc752b21674

• ghcr.io/ufal/atrium-ocr-postprocess-api:1.13.1-beta = sha256:d6e4d68011a0552f6f3cda5aaf2be608eab3ddab475313dae655e86ad56ca2c2

Full Changelog: v1.12.1-beta...v1.13.1-beta

Minor changes and work on issues as pre-pilot preparation

Note

Same code as v1.12.0-beta, whose images were never published (its Docker run stopped at the test step). This release publishes ghcr.io/ufal/atrium-ocr-postprocess:1.12.1-beta and ghcr.io/ufal/atrium-ocr-postprocess-api:1.12.1-beta, and moves latest from 1.11.0-beta.

What's new

• tests/test_categorization_walkthrough.py compares the walkthrough table cell by cell, not its alignment.

What changed since v1.11.0-beta is in the notes of v1.12.0-beta.

Compatibility

The spec changes only its version. Thresholds, weights and batch output are those of v1.12.0-beta.

Full Changelog: v1.12.0-beta...v1.12.1-beta

Pre-production fixes and test added

Production-readiness release. Thresholds, weights and word lists are unchanged and batch output is identical, so DOC_LINE_CATEG_1210 stays valid.

Changed for API callers

• /process and /score_record now run the CPU pre-filter first, as the batch pipeline does. Numeric, code-like and blank lines come back Non-text or Empty (score 0.0, language N/A) without a model call; every other line is scored on its repaired text. Such lines that were answered Clear, Noisy or Trash before change category. A page's score averages Clear, Noisy and Trash lines only.
• PPL_BATCH_LINES defaults to 32. A .env copied from an older .env.example still sets 128.
• GPU work is serialised per process, and GET /health?deep=true fails while no model is loaded.
• Paradata names the perplexity model and layoutlmv3 when they ran, so the recorded licence no longer understates an API run. /score_record takes its page and line caps from the configuration.
• A JSON or text upload without page onto a multi-page record is skipped with a warning instead of landing on page 1.

Reliability

• A perplexity batch that runs out of GPU memory is split and retried; only a line that fails on its own scores 99999. Values are unchanged.
• The ordering of CATEG_TRASH_SCORE_MAX < CATEG_NOISY_SCORE_MAX, SHORT_PPL_CAP < PERPLEXITY_THRESHOLD_MAX and the positive weight sum is checked at startup: the service refuses to start and classify_TEXT.py exits before loading a model.
• Per-document CSVs are written as <doc>.csv.part and renamed when complete, so a failed re-classification keeps the previous file. A leftover .part is safe to delete.
• Exit status is non-zero when run_pipeline.py aborts a stage, when classify_TEXT.py loses its GPU or fails every document, and when page_split.py finds no input directory. Orchestrators that treated these as success will now see them.
• Config files are read as UTF-8, and the Qwen-scale warning no longer fires on every run.

Packaging

• API-only installs list pandas and tqdm, and the torch spec matches the requirements.
• The security scan targets the API image.
• setup/setup_api_server.sh fetches FastText at the Dockerfile's pinned revision and checks its SHA-256.

Documentation and tests (#5)

• docs/categorization_logic.md is rebuilt from the code: order of evaluation, overrides with what blocks each, language tiers, the four document passes, foreign-language limits, worked examples.
• README Step 4 is restructured.
• tests/test_categorization_walkthrough.py follows a line through every step and pins the known limits.

Warning

No container images were published for this tag (its Docker run stopped at the test step). Use v1.12.1-beta: the same code, with images.

Full Changelog: v1.11.0-beta...v1.12.0-beta

Minor refinements + sidecar from David

What's new

• LOWPPL_CLEAR_MAX 50 → 35 (#3). A line of three or more words now takes the language-model shortcut to Clear only below perplexity 35. From 35 to 50 it goes through the score routing like any other line.
  • Over both collections it moves 62,859 lines (0.09 %) in 16,687 documents: Noisy → Clear 37,062, Clear → Noisy 12,342, Noisy → Trash 9,272, Clear → Trash 3,900, Trash → Noisy 279, Trash → Clear 4. The band changes on 1,128 of 1,226,349 pages.
  • On the labelled sets it fixes 63 lines and breaks none: 60 of @david-spacil's and 3 of the 2,067 gold (510 → 507 errors). @DanaKriv's lines do not move.
  • This closes the pilot's parameter work. The witness floor and the shape witness stay off.
• No band for a page without text lines (batch path). aggregate_STAT.py gave a page made only of Empty / Non-text lines the band Clear in its document record. It now omits quality_band there, as the service already did. In the July collections that was 48,544 pages (4 %). The page-stats CSV has no band column and is unchanged.
• tools/corpus_change.py (#3, stages 16–18):
  • reach re-scores whole collections in memory under several configurations and writes only the changed lines, with labels joined and page aggregates (--pages, --page-ref). pagecheck applies the stage-17 rule to its summary.
  • recat re-scores a collection into a new folder, sharded and resumable (--expect-missing).
  • diff compares two re-scored trees, lines and pages, with an independent recount (--verify). It pairs lines by (page_num, line_num), so a document stored in another order is counted, not refused.
• tools/ab_constant_eval.py (both copies): the exact McNemar test no longer overflows when more than 1,023 lines disagree.
• Labels (tools/gold/, see GOLD.md):
  • sidecars/floor_2026-10-05_keys.csv: @david-spacil's floor annotation, keys and labels only, on 141,292 lines of the July batch;
  • one gold label corrected: sektlll (CTX201703691 149:287) is Noisy, not Trash.
• Docs: README.md on what avg_quality_score leaves out, and docs/categorization_logic.md on the new threshold. The pre-commit size limit is raised to 6,024 KB for the sidecar.

Compatibility

• Categories change for lines of three or more words with a perplexity from 35 to 50, and, through the page rules, for a few lines next to them. Re-scoring a stored batch with this version moves about 0.09 % of its lines.
• In document records from the batch path, pages without text lines no longer carry quality_band. Nothing else in records, the API or openapi.json changes.

The ARUP and ARUB collections are re-scored with this version as DOC_LINE_CATEG_1210, with page stats (#3, stage 18). They are delivered separately, not as release assets.

Full Changelog: v1.10.0-beta...v1.11.0-beta

Edits for shared scripts and quality summary

What's new

• quality_summary in the record (ufal/atrium-project#73 R6). Every write of pages or lines (/process, /score_record, batch stages) sets page and line totals, the mean, median, min and max of the pages' quality_score, and the lines per categ. Numbers only: nothing routes or refuses on it.
• A malformed ALTO upload is a 422 (cause: malformed) instead of an empty 200.
• tools/corpus_change.py measures a categorisation arm over whole collections (#3, stage 16).
• Tests for the W4 OCR hand-off and the port guard; shared modules of 9 October.

Compatibility

Additive for records. A client that sent broken ALTO and accepted an empty result now gets a 422.

Full Changelog: v1.9.1-beta...v1.10.0-beta

API edits for digital-convert repo

A page that digital-convert cannot vouch for now comes back into the same record (atrium-digital-convert#4 W4). digital-convert flags such a page needs_ocr, the pipeline runs ATR on it, and this service merges the page's ALTO into digital-convert's record: that page only, with every born-digital page left as it was. The spec stays compatible: every change is additive.

POST /process: the OCR hand-off (atrium-digital-convert#4 W4)

• In: the ATR ALTO of one page (file), digital-convert's record (document_json), and page, a new optional form field: the record's own key for that page (ii, A-1, …).
  • Without page, the ALTO's PHYSICAL_IMG_NR is read as a physical position and mapped through the record's pages[].page_index. A number is never matched to a label.
• Refused before any model runs (422):
  • a page the record lacks, or does not flag needs_ocr;
  • a multi-page ALTO with page;
  • page without a record, or with a document upload;
  • page on a born-digital record that flags nothing.
• Written into the record:
  • the page's lines[] are replaced by the classified lines (text, categ, quality_score, lang), in the record's page order;
  • its pages[] row gets quality_score, quality_band and ocr.engine (the engine the ALTO names, e.g. ocr:pero, else the configured ALTO origin);
  • its page_index, canvas, needs_ocr, text_layer and category stay;
  • source, content and every other page are untouched.
  • Both stamps read contribution: "ocr-handoff".
• Out: the usual response, plus ocr_handoff: page, lines_replaced, lines_written.
• A JSON or text upload names no page of its own: on a record that flags pages, it needs page, and without it nothing is written.
• Chain the calls: send each flagged page with the record the previous call returned.

Record writes (document_hook.write_document_block())

• On a record that flags pages:
  • only those pages' rows are written, keyed by the record's own labels (handoff_pages, record_page_key);
  • their lines are replaced;
  • a page row is limited to its quality and ocr (HANDOFF_PAGE_FIELDS);
  • content, tables and rows of other pages are held back, with one warning.
• The batch stages make the same write.
• A source whose origin names another originator than the record's is no longer written onto it. An OCR of the original's pages is another file, and it used to cause conflict warnings.

Shared modules (hub-canonical, re-vendored)

• atrium_document.py:
  • the hand-off is per page: a row on a page the record does not flag is refused, and so is a wholesale set_block();
  • DocumentRecord.replace_page_rows();
  • OCR_HANDOFF, which merge_document_records() accepts while a page is flagged.
• Record schema (still 1.0, additive): pages[].text_layer, a closed enum, written by digital-convert.
• atrium_service.py: the reason code source_digest_mismatch (422), now in every spec.
• service/openapi.json regenerated:
  • /process gains page;
  • ProcessResponse gains ocr_handoff (OcrHandoff);
  • the record schema gains text_layer.

Upgrade notes

• Additive: an existing call behaves as before, with two exceptions:
  • an ALTO sent with a born-digital record that flags pages: its page is now written, page by page, where before rows were added under the ALTO's own label;
  • an OCR file's source is no longer written onto a record whose source describes another file.
• To re-acquire a page, pass its key as page. The PHYSICAL_IMG_NR mapping covers ALTO that numbers pages by their physical position.

Assets

service_bundle.zip, openapi.json and openapi.json.sha256 (sha256sum -c openapi.json.sha256).

Full Changelog: v1.9.0-beta...v1.9.1-beta

API change for digital-convert repo + minor algo and param optim edits

Born-digital lines get the same quality answer as OCR lines (atrium-digital-convert#4, W3): the common line-quality model now scores an existing record in place. The release also carries the #3 parameter work since v1.8.0-beta. The spec stays compatible: /score_record is a new endpoint, and nothing existing changed.

POST /score_record (atrium-digital-convert#4 W3)

• In: document_json (required; any record, born-digital or OCR) and an optional pages: the record's own page keys, comma-separated (i,ii,3). A key the record lacks is 422.
• Scored: each lines[] row with text, exactly as it is: no re-reading, splitting or reordering, so every answer maps back onto its row. The scorer is the same as on every other path, batched by PPL_BATCH_LINES.
• Skipped and counted:
  • lines that carry atrium-digital-convert's decode verdict (categ Garbage or Inverted): their text layer does not decode, so the verdict stands;
  • lines without text.
• Written: lines[].categ / quality_score / lang and pages[].quality_score / quality_band, on rows that already exist. Never text, bbox or source, and never a new row.
• Out:
  • cleaned_lines: the AltoLine fields, plus the record's line. page holds the record's page key (e.g. iv), not a 1-based index;
  • pages[]: lines_scored, skipped_decode_verdict, skipped_empty, quality_score, quality_band;
  • limits_applied;
  • document_json: scored, or returned as sent when nothing was scored;
  • paradata: the run's CreateAction.
• Limits: MAX_UPLOAD_MB (the record part), ATRIUM_TEXT_INGEST_MAX_PAGES (pages scored) and ATRIUM_TEXT_INGEST_MAX_LINES_PER_PAGE (lines on one page), each 413 limit_exceeded. A record that cannot be opened, or has no doc_id, is 422 invalid_record.
• atrium-digital-convert's POST /describe calls it when OCR_POSTPROCESS_URL is set. Against an older version of this service, it falls back to /process, one page at a time. Those scores reach the /describe report only; the record is not written.

Record writes

• document_hook.write_scores() is a scoring-only write path:
  • only the scoring fields above; any other field is refused before a file is opened;
  • existing rows only;
  • lines with a decode verdict are left alone;
  • the result is validated before it is written.
• Re-vendored atrium_document.py (SCORING_FIELDS). On a record another tool originated (a born-digital one), such a merge is a scoring co-contribution: noted, not refused, and stamped contribution: "scoring". The record stays the converter's, and the fan-in accepts it. Any other write onto a foreign record is still refused unless a page asks for OCR (needs_ocr). This tool's own OCR records are unchanged.
• DIGITAL_CONVERT_KINDS gains ODT, ODS, XLSX and RTF, which atrium-digital-convert v1.1.0-beta reads with this repository's text_formats.py. This changes only the wording of the text-split report note.

Categorisation (#3)

• Witness floor (rule_short_garbage_witness_floor), new and shipped off. With [TEXT_UTILS] SHORT_GARBAGE_WITNESS_FLOOR = true, a short line the cascade rated Clear becomes Noisy when the shape witness finds garbage in it.
  • It is off in setup/config.txt and in the code, so categories do not change for any input.
  • The stage-13 figures are in the setup/config.txt comment, and docs/categorization_logic.md describes the rule. Switching it on is the maintainers' call.
• tools/ab_constant_eval.py reports gold-Clear lines sent to another class (Clear-demoted). Its discordant dump gains ref_pred / arm_pred and clear_loss rows. Developer tools only.

Note

setup/setup_api_server.sh now uses the virtualenv directory venv-ocr (it was venv-alto). An existing installation gets a fresh environment on the next run.

Assets

service_bundle.zip, openapi.json and openapi.json.sha256 (sha256sum -c openapi.json.sha256).

Full Changelog: v1.8.0-beta...v1.9.0-beta