v0.2.0

Restore correctness release. Several format fixes remove failure modes that
could make a printed document unrecoverable, and a measurement campaign settled
the long-open question of whether trained OCR models are worth shipping (they
are not).

Breaking (pre-1.0, no compatibility shim): the wire format changed. The
identifier stays v1 (magic B1, _VERSION = 1, codec names unchanged), but
documents produced by 0.1.0 do not decode with 0.2.0. Re-create any archive you
intend to keep.

Highlights

• No trained OCR model is needed for any codec. base16g never required
  one; base32g no longer does either, because this release's format fixes
  moved it from model-required to stock-viable. The published
  glyphive-ocrmodel-* packages were trained on the wrong data and are not
  recommended for anything.
• base32g is Courier-only on stock OCR (measured 4–10pt across 3 fonts):
  it fails on OCR-B and DejaVu Sans Mono at every size. Documented so nobody
  picks it blind.
• A very short final data line no longer decides whether a document
  restores. This was a lottery every document ran at create time.

Added

• glyphive train (experimental) — build an OCR model for a codec/font/size
  with the data-integrity gates that every previous attempt lacked: it verifies
  that each row image actually shows its paired text, aborts on any unencodable
  transcription or non-zero trainer skip ratio, derives a narrowed unicharset
  from the codec registry instead of inheriting the base model's ~112
  characters, and builds its own starter artifacts. Reported CER is labelled a
  proxy everywhere it appears; a produced model records gate_verdict=UNGATED
  because character error rate has repeatedly failed to predict restore. Not
  needed to use glyphive — it exists so future experiments are measurable.
• Repo-resident E2E benchmark grid harness (benchmarks/e2e_grid.py):
  create → rasterize → OCR → restore over a font-size × line-width × codec ×
  line-parity grid. Per-cell status is one of
  restored/not-restored/not-built/error; a configuration create
  refuses to build is excluded from every restore-rate denominator (reported
  as n/m testable (k not built), or a loud UNTESTED), --repeat N
  aggregates across repeat documents, the corpus is a pinned checked-in
  fixture, and results JSON carries full provenance (commit, engine version,
  corpus digest). Harness logic is unit-tested with a fake OCR provider, no
  engine required.
• Physical-scan regression fixture + gate
  (tests/fixtures/physical_scan/, wired into tests/test_ocr_transcripts.py):
  a committed, deterministically degraded page render (Gaussian blur 0.6 +
  0.7° rotation, generation script included) restored byte-for-byte through
  extract --from-images with tesseract-glyphive; skips cleanly without
  the engine. Guards the real-scan path with zero real user content.
• Per-glyph deletion stats in tools/ocr_font_report.py
  (align_and_tally): aligned-diff tallies of per-glyph drop rate and
  insertion adjacency alongside the existing confusion stats — OCR dropping
  a thin glyph shrinks the line and desynchronizes the fixed-width frame
  parse, a strictly worse failure than a substitution, so candidate alphabets
  are now gated on both.

Fixed

• A very short ("runt") final data line no longer decides restore.
  Whether a document restored at small font sizes depended on
  encoded_length mod line_capacity: a few-character final data line is
  destroyed by OCR page segmentation (and can corrupt the line above it), so
  one payload byte more or less flipped restore between OK and FAIL
  (measured: 13-char final line fails, 153-char passes —
  benchmarks/results/fourpt-runt-line-20260721.json). The codec now
  zero-pads the protected stream so the final data line's printed payload is
  never below half the line width; the header's recorded length stays
  unpadded, so decode truncates the pad with no decoder change.
• Group-packed decode no longer crashes on an out-of-range group. For
  codecs where radix**group_chars > 256**group_bytes (basemaxg, base85,
  z85) an OCR misread could decode a group past its byte range;
  int.to_bytes then raised OverflowError, which sailed past every
  ValueError erasure handler and aborted the whole restore. Such a group
  now raises ValueError and is absorbed as an ordinary erasure.
• --descan auto-retry no longer re-OCRs the sharp pass. The failure-path
  retry ladder re-ran radius 0.0 over every page even though the initial
  sharp sweep was already computed; the retry now OCRs only the additional
  radii (0.6, 0.8) and merges onto the existing sharp results as the ordered
  spine — one full OCR sweep saved per retried document, byte-identical
  outcome.
• tools/ocr_font_report.py whitelist quoting: an alphabet containing a
  shell-special character (e.g. an embedded ") crashed pytesseract's
  internal shlex.split before OCR ran; the whitelist config is now
  shell-quoted.

CI

• PyPI publish and GitHub release no longer depend on the docs deploy.
  A GitHub Pages/docs failure blocked the v0.1.0 package publish twice;
  docs-build/docs-deploy are now a parallel, best-effort branch off the
  release-critical path.

Added (earlier in this cycle)

• OCR per-character confidence drives char-level erasure marking (plan
  3). A CRC-failed line used to erase its ENTIRE byte span for the
  document-level Reed-Solomon tier, even though the typical cause is one or
  two misread characters — the erasure budget was consumed far faster than
  the true error mass. OcrProvider.ocr_image now returns
  List[OcrLine] (text + optional per-character confidence, 0..1 or
  None) instead of List[str] — a breaking change to the provider
  contract, fine pre-1.0. Tesseract builds confidence from
  image_to_data (word-granularity, broadcast per character — Tesseract's
  stable API has no finer level); EasyOCR/PaddleOCR broadcast their own
  per-segment/per-line score. Base16GCodec.decode_spool accepts an
  optional char_conf (keyed by physical line order, since a CRC-failed
  line's own index label may be corrupt): for a failed line with usable
  confidence, only the byte offsets its low-confidence characters map to
  are marked as erasures — the line's other bytes enter the RS stream as
  ordinary (unverified) data. A two-pass, block-local safety valve makes
  this strictly no-worse than today: if a block still fails RS with the
  narrower erasure set, it is retried with the touching line(s) promoted
  to a full-span erasure (today's behaviour) before giving up. This is a
  hint about erasure position only — acceptance is still CRC/RS/SHA-256,
  never guessed. char_conf absent (the default) is byte-identical to a
  build without this feature. New tools/conf_calibration.py measures
  P(char wrong | conf < t) and recall of wrong characters against
  Tesseract to calibrate the default threshold (ships at 0.6 pending a
  real calibration run — see the tool's own docstring).

Fixed (earlier in this cycle)

• Default --line-parity (2) broke restore on the primary OCR path
  (breaking, pre-release — no compat shim). The constrained Tesseract
  character whitelist used for scanning strips ALL interior spaces from
  every printed line, producing the "compact frame" form
  split_frame/split_frame_with_parity already tolerate. But the reader
  determined the printed per-line Reed-Solomon field's width by counting
  whitespace tokens per line (_detect_line_parity_chars), and a
  one-token compact line cannot vote — every line silently fell back to
  width 0, folding the line-parity characters into the payload and
  corrupting the byte count for the whole stream. At 3pt OCR-B end-to-end,
  --line-parity 0 round-tripped fine while --line-parity 2 (the
  default) and 4 both failed with cannot recover RS parameters: data/parity line counts are inconsistent or a per-line CRC/RS-budget
  error — the default was worse than off. The protected machine header
  (layout.py) now carries nsym_line as an authoritative field
  (_machine_header_bytes/_decode_machine_header gain one byte); the
  restore path (restore/decode.py) reads it from the CRC/RS-protected
  header — decoded before any payload line is even classified — and
  passes it straight to RadixCodec.decode_spool, which prefers it over
  the token-counting heuristic. _detect_line_parity_chars remains the
  fallback for headerless/raw-codec callers (e.g. glyphive inspect's
  read-only describe_line_stream). Existing documents from before this
  change do not decode (the machine header envelope grew one byte).

Changed (earlier in this cycle)

• Page parity lifts the 255-page cap: --parity-pages now supports up to
  65,535 total pages (breaking, pre-release — no compat shim). Document-level
  whole-page recovery (codec/pagers.py) previously required data pages +
  parity pages <= 255 (one Reed-Solomon symbol per byte, GF(2^8)) — too small
  for large archives (a 30 MB tree at ~3 KB/page is ~10,000 pages). Page
  parity now automatically switches to a GF(2^16) field (one symbol per pair
  of bytes) whenever the total exceeds 255, raising the cap to 65,535; the
  protected machine header gains a pgpar_field byte (8 or 16) recording
  which field a document uses, and glyphive inspect reports it. GF(2^8)
  documents (<=255 total pages) are unaffected byte-for-byte. Existing
  documents from before this change do not decode (the machine header
  envelope grew one byte to carry the new field).

• Wire format hardened: interleaved parity, kind-covered CRC, optional
  per-line Reed-Solomon (breaking, pre-release — no compat shim). The
  base16g-crc16-rs codec (and every denser radix codec sharing its
  engine) fixes three defects found while measuring recovery under a
  substitution-error channel, at zero size cost for the first two:
  (1) parity byte interleave — the document-level Reed-Solomon parity
  stream is now written symbol-major (parity byte j of block b at
  j*nblocks+b instead of b*nsym+j), so one corrupted parity LINE
  spreads its damage across every block instead of wiping one block's entire
  parity budget outright; (2) kind-covered CRC — the per-line check field
  (and the H/T/Q machine frames in layout.py) now covers the leading
  L/P/H/T/Q kind letter, so a misread that flips one kind
  into another now fails its own CRC instead of silently producing a
  CRC-valid phantom line; (3) optional per-line Reed-Solomon — each
  printed line may now carry nsym_line (0, 2, or 4; default 2, exposed as
  create --line-parity) extra parity bytes over its own index token and
  payload, self-healing many single/double-character OCR errors in place
  before they ever touch the document-level RS erasure budget. The group
  header grew one byte (B1 | version | nsym | nsym_line | orig_len,
  8 → 9 bytes) to record the new field; existing documents from before this
  change do not decode. Measured effect: the recovery cliff (channel_sim,
  30 KB docs, 12% document parity) moves from failing at 0.1% substitution
  error to succeeding at 0.5% at the nsym_line=2 default.

Fixed (earlier in this cycle)

• Decode hardening (rescues documents that previously hard-failed). A
  CRC-failed line whose index token was misread used to be trusted for stream
  geometry, so one bad character could claim an impossible index (~900,000 vs a
  true max ~1,000) and make decode die with "cannot recover RS parameters" even
  with the parity budget barely touched — every ~30 KB document failed at 0.1 %
  character error. Decode now (1) attempts a CRC-guided single-substitution
  repair of each failed line (accepted only when exactly one candidate
  reproduces the printed CRC — the CRC is the oracle, never decompressibility),
  (2) computes stream geometry only from CRC-valid lines and positionally
  reassigns or drops implausible-index failed lines, and (3) degrades a
  conflicting-duplicate collision to an erasure (which Reed-Solomon rebuilds)
  instead of aborting the whole decode. Applies to all existing documents; no
  bytes on paper change, and the clean-transcript fast path is untouched.

Full Changelog: https://github.com/jose-pr/glyphive/commits/v0.2.0