GOT-OCR2.0 int8 weights (models-got-ocr2-v1)

GOT-OCR2.0 frankenized int8 .focrq (776MB) + qwen.tiktoken tokenizer. Apache-2.0 (StepFun). Pull with: focr pull got-ocr2

v0.3.0 — first-run UX (model auto-resolve + -o/--output) + PDF hardening

Pure-Rust, CPU-only runner for the Baidu Unlimited-OCR model. Install:

curl -fsSL https://raw.githubusercontent.com/Dicklesworthstone/franken_ocr/main/install.sh | bash
focr pull            # fetch the int8 weights once (~3.9 GB)
focr ocr page.png    # or: focr ocr page.png -o page.json

Prebuilt focr binaries for macOS (arm64 + x86_64), Linux (x86_64 + aarch64), and Windows (x86_64). Each asset has a .sha256 sidecar. Weights are unchanged from v0.2.0 (focr pull fetches them).

---

The first-run-experience release. A clean-machine install report exposed three
papercuts on the path from "curl the installer" to "OCR a page", and all three are
now closed: the installer no longer aborts under gum, a freshly-pulled model is
found with no flags, and focr ocr can write its result straight to a file —
including structured JSON with bounding boxes. The native PDF path picked up two
robustness fixes on top: a decompression-bomb bound and a machine-visible per-page
skip event. Still pure, memory-safe Rust — no Python, no CUDA, no FFI at inference.

Added

• focr ocr -o/--output FILE (bd-sreb). Writes the OCR result to a file
  instead of stdout; the format follows the extension — .json emits structured
  JSON, any other extension (e.g. .md) emits markdown, and --json forces JSON.
  The structured JSON carries the rendered markdown plus a layout array of
  {label, boxes} spans, each box [x1, y1, x2, y2] in source-image pixels (a PDF
  nests these under a per-page pages array). The same shape is what --json
  prints to stdout. New engine entry points OcrEngine::recognize_with_layout /
  recognize_dynamic_with_layout (and _model variants) return the markdown and
  the layout parsed from the same decode, so the two can never disagree
  (e3c3e68).
• Robot per-page skip event for PDFs (bd-fck1). The resilient PDF document
  loop used to drop an undecodable page silently from the machine stream. It now
  emits a structured page NDJSON event (status=skipped, 1-based page number, a
  machine-classifiable error_kind) in robot mode, so a consumer can tell the
  document is missing pages; human mode keeps the stderr warning. page is an
  already-advertised event kind, so the robot schema stays v1
  (ba71ff4).

Fixed

• Fresh-install OCR happy path (bd-3u6x, critical). focr pull installs the
  model as unlimited-ocr.int8.focrq, but the default focr ocr lookup previously
  searched only the bare unlimited-ocr.focrq basename — so a freshly-pulled model
  was invisible without a manual --model, and the clean-machine happy path was
  broken. The resolver now also probes the quant-suffixed names focr pull
  installs (.int8.focrq, .int4.focrq); a pulled model resolves with no flag. An
  exact-basename match still wins over a quant variant
  (e3c3e68).
• Installer aborted instantly under gum (bd-1km0). The first status line
  rendered gum style … "-> …", so gum parsed the leading -> as an unknown
  flag, printed its usage, and under set -euo pipefail aborted the whole install.
  It only triggered with gum installed AND an interactive TTY, so no
  non-interactive CI run ever exercised it. The whole set -e/arg-parse class is
  fixed: every gum style text arg gets a -- flag terminator; check_disk_space
  guards a df that exits non-zero when the default ~/.local/bin parent does not
  exist yet on a fresh account; the checksum pipelines and a failed install
  (ETXTBSY on reinstall-while-running) are guarded; the lock path is per-user and
  $TMPDIR-aware; install.ps1 null-guards the top-level LOCALAPPDATA /
  USERPROFILE Join-Path so non-Windows pwsh reaches the friendly message, not
  a stack trace; and a FOCR_INSTALL_BASE_URL override lands for mirrors / airgap /
  tests. A new true end-to-end installer test (tests/installer_e2e.sh, wired into
  scripts/check.sh + CI with gum installed) drives the real installer
  through the gum/pty path against a fake file:// release into a fresh-account
  dir — so this class of bug can never ship silently again
  (7f25594,
  89c5b21).

Security

• PDF decompression-bomb bound (bd-2zpu, bd-2yqe). lopdf's
  Stream::decompressed_content materializes the full inflated output before any
  length check, so a tiny highly-compressed sole-FlateDecode image stream (a "zip
  bomb") could inflate to gigabytes and OOM the process. decompressed_stream now
  inflates a sole FlateDecode (no PNG/TIFF predictor) itself via the pure-Rust
  flate2 ZlibDecoder + Read::take(cap + 1) under an expected_sample_cap
  bound (4× the already-MAX_PIXELS-bounded declared sample bytes); a clean inflate
  that overruns the cap is the bomb signal and errors, while a non-zlib raw stream
  falls back to lopdf's framing-tolerant decoder. A fresh-eyes follow-up
  (bd-2yqe) clamped a hostile declared bit-depth that could otherwise saturate the
  cap to u64::MAX and reopen the hole. flate2 is a direct dependency now
  (already in the lock graph via lopdf/png/tiff; pure-Rust miniz_oxide
  backend, no FFI)
  (ba71ff4,
  89c5b21).

v0.2.0 — native PDF OCR, robot run_complete carries markdown, native Windows

franken_ocr v0.2.0 — the document-input release

focr now reads PDFs natively, the agent-facing robot stream finally carries the recognized text, and native Windows (x86_64) is proven end to end — all in pure, memory-safe Rust, still with no Python, no CUDA, and no FFI at inference.

Added

• Native PDF page OCR — focr ocr file.pdf (bd-0a7). Scanned / image-XObject PDF pages are rasterized in-process, in pure memory-safe Rust with no FFI, then fed through the identical preprocess → vision → decoder → postprocess pipeline a PNG takes — no out-of-band pdftoppm. The fast path covers JPEG (DCTDecode), CCITT Group 4 fax, and FlateDecode/LZWDecode/ASCII85Decode raw rasters in RGB / gray / CMYK / 1-bpc bilevel, with /Rotate applied. Pages render lazily one at a time. JPXDecode (JPEG 2000), JBIG2Decode, and born-digital vector/text pages (no pure-Rust decoder) surface a precise, actionable error instead of a wrong guess. The document loop is per-page resilient: an undecodable page is skipped so one bad page never discards the rest, while whole-run conditions (model-not-found, cooperative cancel, model format mismatch) abort immediately.
• Native Windows (x86_64) (bd-3u97). focr.exe runs full OCR end to end on Windows 10; robot selftest passes 24/24 (incl. the K=6848 i32-overflow case), focr pull completes the full 3.9 GB multi-part download over the native async HTTP/TLS stack (the WSAENOTCONN send-path bug, bd-15ow, is fixed), and install.ps1 installs + offers to pull. ARM64 Windows is not yet published.

Changed

• Robot run_complete carries the recognized markdown (bd-3o5p). focr ocr --robot / focr robot run previously emitted a payload-less run_complete, so a machine consumer received no OCR text at all; the terminal success event now carries the markdown. No schema-version bump (run_complete was already an advertised event kind).
• Input is now document images and PDFs, not image-only.

Fixed and hardened (PDF codec review)

A two-pass fresh-eyes security/correctness review of the new PDF path before release:

• 1-Gpx declared-dimension guard (computed in u64) fires before any per-pixel allocation — closes a gigapixel-claim overflow/DoS.
• CCITT no longer pre-reserves from the attacker-controlled /Height.
• Multi-filter codec chains ending in DCTDecode/CCITTFaxDecode are rejected with an accurate message instead of feeding still-encoded bytes to the codec.
• Cooperative cancel (Ctrl+C) and model format-mismatch propagate instead of being swallowed per page.

Performance

• The ~9.9 MB BPE tokenizer is cached on the model (OnceLock) — a multi-page PDF parses tokenizer.json once instead of twice per page.

---

Binaries are raw single-file executables (one per architecture, ISA dispatched at runtime), each with a <asset>.sha256 sidecar. Linux is gnu (glibc), not musl. Fetch the weights once with focr pull (~3.9 GB int8, SHA-256 verified) and run fully offline thereafter.

Full history: CHANGELOG.md. Model weights remain MIT, Copyright (c) 2026 Baidu.

v0.1.0 — first release

First tagged release of franken_ocr / focr — a pure-Rust, CPU-hyper-optimized runner for the Baidu Unlimited-OCR model (no ML framework).

Highlights

• Working end-to-end OCR — CER 0.0094 vs the Baidu baseline on a real page (beat the oracle on one token).
• int8 decode path + runtime ISA auto-dispatch: SDOT/SMMLA on ARM, AVX2/AVX-VNNI/AVX-512-VNNI on x86 — one binary picks the best tier your CPU has.
• focr convert — safetensors bf16 → int8 .focrq (byte-identical to load-time int8, proven on the real 6.67 GB model).
• focr pull — auto-downloads the weights in the optimal format over the native asupersync HTTP stack (rustls/webpki-roots), verifies every byte by sha256, caches for fully-offline inference. Production-verified against this repo's models-v1 release + Hugging Face.
• focr robot selftest — verifies the dispatched int8 kernels are bit-exact on your silicon (proven on Apple SDOT + Threadripper 5995WX AVX2, 24/24).

Get the model

focr pull          # downloads + verifies the int8 weights (~3.9 GB) into the cache
focr ocr page.png  # then runs fully offline

Binaries

Each focr is a single portable binary that auto-detects your CPU and dispatches to the best int8 kernel at runtime — so one x86_64 binary covers AVX2 / AVX-VNNI / AVX-512-VNNI, and one aarch64 binary covers NEON / SDOT / SMMLA-i8mm. No per-CPU-feature variant to choose.

Platform	Asset
macOS — Apple Silicon (M-series)	focr-aarch64-apple-darwin-neon-sdot-i8mm
macOS — Intel	focr-x86_64-apple-darwin
Linux — x86-64 (Intel/AMD, incl. Threadripper)	focr-x86_64-unknown-linux-gnu
Linux — ARM64 (AWS Graviton, etc.)	focr-aarch64-unknown-linux-gnu

# example: Linux x86-64
base=https://github.com/Dicklesworthstone/franken_ocr/releases/download/v0.1.0
curl -fLO $base/focr-x86_64-unknown-linux-gnu
curl -fLO $base/focr-x86_64-unknown-linux-gnu.sha256
shasum -a 256 -c focr-x86_64-unknown-linux-gnu.sha256     # verify integrity
chmod +x focr-x86_64-unknown-linux-gnu
./focr-x86_64-unknown-linux-gnu robot selftest            # confirm int8 kernels are bit-exact on your silicon

Every binary ships a .sha256 sidecar. The x86_64-unknown-linux-gnu build is verified bit-exact against the scalar oracle on real AVX2 hardware (robot selftest 24/24, including the K=6848 i32-accumulation overflow case).

Model weights (int8) — unlimited-ocr

int8 .focrq weights for franken_ocr (Baidu Unlimited-OCR, MIT).

Reassembled unlimited-ocr.int8.focrq (3,914,093,440 B, sha256 d8c5fcf223c8e062af63f6b86964d099e2c5a5b272ae096a09433aaf5510a440) ships as two sub-2GB parts that focr pull reassembles + sha256-verifies:

• unlimited-ocr.int8.focrq.part00 — sha256 e58503fb1700a56cff71d2c136c223efa5c10c38604901987904467267c815e3
• unlimited-ocr.int8.focrq.part01 — sha256 6a647a04fece7bced666c0e56581969c1a799e8447c64e9abf37ee10c322ec7d
• tokenizer.json — sha256 a02f8fd5228c90256bb4f6554c34a579d48f909e5beb232dc4afad870b55a8b4

Run focr pull (uses models/manifest.json) to fetch + verify, then focr ocr page.png. See models/UPLOAD_RUNBOOK.md.