API edits for digital-convert repo

A page that digital-convert cannot vouch for now comes back into the same record (atrium-digital-convert#4 W4). digital-convert flags such a page needs_ocr, the pipeline runs ATR on it, and this service merges the page's ALTO into digital-convert's record: that page only, with every born-digital page left as it was. The spec stays compatible: every change is additive.

POST /process: the OCR hand-off (atrium-digital-convert#4 W4)

• In: the ATR ALTO of one page (file), digital-convert's record (document_json), and page, a new optional form field: the record's own key for that page (ii, A-1, …).
  • Without page, the ALTO's PHYSICAL_IMG_NR is read as a physical position and mapped through the record's pages[].page_index. A number is never matched to a label.
• Refused before any model runs (422):
  • a page the record lacks, or does not flag needs_ocr;
  • a multi-page ALTO with page;
  • page without a record, or with a document upload;
  • page on a born-digital record that flags nothing.
• Written into the record:
  • the page's lines[] are replaced by the classified lines (text, categ, quality_score, lang), in the record's page order;
  • its pages[] row gets quality_score, quality_band and ocr.engine (the engine the ALTO names, e.g. ocr:pero, else the configured ALTO origin);
  • its page_index, canvas, needs_ocr, text_layer and category stay;
  • source, content and every other page are untouched.
  • Both stamps read contribution: "ocr-handoff".
• Out: the usual response, plus ocr_handoff: page, lines_replaced, lines_written.
• A JSON or text upload names no page of its own: on a record that flags pages, it needs page, and without it nothing is written.
• Chain the calls: send each flagged page with the record the previous call returned.

Record writes (document_hook.write_document_block())

• On a record that flags pages:
  • only those pages' rows are written, keyed by the record's own labels (handoff_pages, record_page_key);
  • their lines are replaced;
  • a page row is limited to its quality and ocr (HANDOFF_PAGE_FIELDS);
  • content, tables and rows of other pages are held back, with one warning.
• The batch stages make the same write.
• A source whose origin names another originator than the record's is no longer written onto it. An OCR of the original's pages is another file, and it used to cause conflict warnings.

Shared modules (hub-canonical, re-vendored)

• atrium_document.py:
  • the hand-off is per page: a row on a page the record does not flag is refused, and so is a wholesale set_block();
  • DocumentRecord.replace_page_rows();
  • OCR_HANDOFF, which merge_document_records() accepts while a page is flagged.
• Record schema (still 1.0, additive): pages[].text_layer, a closed enum, written by digital-convert.
• atrium_service.py: the reason code source_digest_mismatch (422), now in every spec.
• service/openapi.json regenerated:
  • /process gains page;
  • ProcessResponse gains ocr_handoff (OcrHandoff);
  • the record schema gains text_layer.

Upgrade notes

• Additive: an existing call behaves as before, with two exceptions:
  • an ALTO sent with a born-digital record that flags pages: its page is now written, page by page, where before rows were added under the ALTO's own label;
  • an OCR file's source is no longer written onto a record whose source describes another file.
• To re-acquire a page, pass its key as page. The PHYSICAL_IMG_NR mapping covers ALTO that numbers pages by their physical position.

Assets

service_bundle.zip, openapi.json and openapi.json.sha256 (sha256sum -c openapi.json.sha256).

Full Changelog: v1.9.0-beta...v1.9.1-beta