A comprehensive, up-to-date guide to OCR in the VLM era — models, benchmarks, deployment patterns, and practical decision guides.
Current model rankings — check these first to see what's state-of-the-art before choosing a model.
| Benchmark | What it measures | Link |
|---|---|---|
| OCR Arena | Community arena — test and vote on VLMs & open-source models side-by-side on your own documents | ocrarena.ai ↗ |
| IDP Leaderboard | Comprehensive IDP evaluation: KIE, VQA, OCR, classification, long docs, table extraction, confidence scoring (22+ models) | idp-leaderboard.org ↗ |
| OCRBench v2 | VLM OCR across 10 capabilities (text rec, scene, doc, tables, math) | HF Space ↗ |
| OCRBench v1 | Original VLM OCR evaluation, widely cited | HF Space ↗ |
| olmOCR-Bench | PDF parsing quality via 7k unit tests (tables, math, layout) | HF Dataset + scores ↗ |
| OmniDocBench | Diverse PDF parsing — 9 doc types, layout + formula + handwriting | HF Dataset ↗ |
OCR (Optical Character Recognition) has gone through three distinct eras. Understanding which era a tool belongs to is the first step to choosing the right one.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Era 1 — Classical (1960s–2010s)
─────────────────────────────────────────────────────────────────
Image → Binarize → Segment → Feature Engineering → Rule Engine → Text
Tools: Tesseract v3, ABBYY classic, OCRAD, OmniPage
Good for: clean scans of typed text, legacy pipelines
Era 2 — Deep Learning (2012–2022)
─────────────────────────────────────────────────────────────────
Image → CNN → LSTM → CTC Decoder → Text
Tools: EasyOCR, PaddleOCR, Tesseract v5, TrOCR, Surya
Good for: scene text, multilingual docs, handwriting
Era 3 — VLM (2023–present) ◄ Where most new work happens
─────────────────────────────────────────────────────────────────
Document → Vision Encoder ──► Language Model → Text + Structure
└──► Layout + Tables + Fields
Tools: GOT-OCR, Nanonets-OCR, Mistral OCR, Claude, GPT-4o, Qwen-VL
Good for: complex documents, tables, forms, mixed content
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Answer three questions to narrow down your options:
flowchart TD
A([Start]) --> B{Content type?}
B -->|"Documents<br/>PDFs · scans · forms"| C{"Need structure?<br/>Tables · fields · layout"}
B -->|"Scene text<br/>Photos · signs · labels"| G{Deployment?}
B -->|Handwriting| J{Type?}
B -->|"Math · code<br/>specialised content"| M([Mathpix · Specialized VLMs])
C -->|Yes| D(["VLM-based<br/>GOT-OCR · Nanonets · Mistral OCR<br/>Claude · GPT-4o"])
C -->|"No — plain text only"| E{Constraints?}
E -->|"Privacy / cost<br/>self-hosted"| F(["Open source<br/>Surya · Tesseract · EasyOCR<br/>DocTR · PaddleOCR"])
E -->|"Max accuracy<br/>cloud OK"| DA(["Cloud API<br/>AWS Textract · Azure · Google Vision"])
G -->|"Real-time · mobile"| H(["On-device<br/>Apple Vision · ML Kit<br/>TFLite models"])
G -->|"Batch / server"| I(["CRAFT + PARSeq<br/>DBNet + CRNN · PaddleOCR"])
J -->|"Modern cursive"| K(["VLM · HTR-Flor<br/>TrOCR · EasyOCR"])
J -->|"Historical manuscripts"| L(["Kraken · Calamari<br/>specialised HTR"])
| Constraint | Recommended path |
|---|---|
| Privacy / on-prem required | Open-source self-hosted: Surya, Tesseract, PaddleOCR, EasyOCR, DocTR |
| Tight budget (< $0.001/page) | Open-source models or batch cloud tiers |
| Highest possible accuracy | VLM-based APIs (Mistral OCR, Nanonets, Claude, GPT-4o) |
| Low latency (< 500ms) | Edge models or fast cloud APIs (Google Vision, Apple Vision) |
| Many languages (> 50) | EasyOCR (80+), PaddleOCR, multilingual VLMs |
| Tables & structured data | AWS Textract (AnalyzeDocument), GOT-OCR, Nanonets, Mistral OCR |
| Handwriting at scale | TrOCR, HTR-Flor, or VLM-based (Claude, GPT-4o) |
Open source Cloud API
──────────────────── ─────────────────────────
✓ No per-page cost ✓ No infra to manage
✓ Data stays on-premise ✓ State-of-the-art accuracy
✓ Fully customisable ✓ Scales instantly
✗ You own the infra & ops ✗ Cost grows with volume
✗ Accuracy gap on hard docs ✗ Data leaves your network
Hybrid: use open source for simple pages, cloud API for hard ones.
→ See comparisons/open-source-vs-api.md
awesome-ocr-2026/
│
├── docs/ ← Start here if you're new to OCR
│ ├── 01-history-of-ocr.md
│ ├── 02-how-ocr-works.md
│ ├── 03-ocr-taxonomy.md
│ ├── 04-document-vs-scene-ocr.md
│ ├── 05-vlm-and-multimodal-ocr.md ← The paradigm shift explained
│ ├── 06-layout-structure-and-kie.md ← Tables, forms, KIE
│ ├── 07-handwriting-ocr.md
│ ├── 08-multilingual-ocr.md
│ ├── 09-benchmarks-and-metrics.md
│ ├── 10-costs-latency-and-scaling.md
│ ├── 11-failure-modes-and-evaluation.md
│ └── 12-future-of-ocr.md
│
├── models/
│ ├── pre-deep-learning/ ← Classical & statistical methods
│ │ ├── template-matching/
│ │ ├── feature-based/ (Tesseract v1-v3, Cuneiform)
│ │ ├── statistical-sequence/ (HMM, FSM-based)
│ │ ├── classical-handwriting/
│ │ ├── layout-analysis/ (XY-Cut, Voronoi, Docstrum)
│ │ └── commercial-systems/ (ABBYY classic, OmniPage)
│ │
│ ├── deep-learning/ ← Neural network models
│ │ ├── cnn-lstm-ctc/ (EasyOCR, Tesseract v5, Kraken)
│ │ ├── cnn-attention/ (PaddleOCR, DocTR, MMOCR)
│ │ ├── transformer-based/ (TrOCR, SVTR, MGP-STR)
│ │ ├── end-to-end/ (Surya, PPOCRv4-v5)
│ │ └── handwriting/ (HTR-Flor, SimpleTR)
│ │
│ ├── scene-text/ ← Text in natural images
│ │ ├── detection/ (CRAFT, DBNet, EAST, PSENet)
│ │ ├── recognition/ (CRNN, ASTER, ABINet, PARSeq)
│ │ └── end-to-end/ (FOTS, Mask TextSpotter)
│ │
│ ├── vlm-based/ ← The current frontier
│ │ ├── general-purpose/ (InternVL, Qwen-VL, MiniCPM-O)
│ │ └── ocr-specialised/ (GOT-OCR, Nanonets, OlmOCR,
│ │ DeepSeek-OCR, Mistral OCR,
│ │ PaddleOCR-VL, Ocean-OCR)
│ │
│ └── cloud-apis/ ← Managed services
│ ├── aws-textract.md
│ ├── azure-ocr.md
│ ├── google-vision.md
│ ├── mistral-ocr.md
│ ├── anthropic-claude.md
│ ├── openai.md
│ ├── google-gemini.md
│ ├── mathpix.md
│ ├── nanonets.md
│ ├── abbyy.md
│ ├── apple-vision.md
│ └── baidu-ocr.md
│
├── benchmarks-and-evaluation/
│ ├── metrics/ (CER, WER, ANLS, TEDS, F1)
│ ├── document-benchmarks/
│ ├── scene-text-benchmarks/
│ ├── vlm-benchmarks/
│ └── leaderboards/
│
├── use-cases/ ← Implementation guides
│ ├── document-ocr.md
│ ├── invoice-extraction.md
│ ├── id-document-ocr.md
│ ├── scene-text-and-mobile.md
│ ├── ocr-to-rag.md
│ └── hybrid-and-ensemble-ocr.md
│
├── comparisons/ ← Side-by-side decision guides
│ ├── best-for-documents.md
│ ├── best-for-scenes.md
│ ├── best-for-handwriting.md
│ ├── best-for-multilingual.md
│ ├── best-low-cost.md
│ └── open-source-vs-api.md
│
├── deployment/ ← Production guides
│ ├── self-hosting.md (Docker, Kubernetes, queues)
│ ├── cloud-deployment.md (AWS, Azure, GCP)
│ ├── edge-and-mobile.md (iOS, Android, embedded)
│ ├── batch-vs-streaming.md
│ └── monitoring-and-quality.md
│
└── resources/
├── papers.md
├── blog-posts.md
├── talks-and-videos.md
└── communities.md
| I want to… | Start here |
|---|---|
| Understand OCR from scratch | docs/02-how-ocr-works.md |
| Understand the VLM shift | docs/05-vlm-and-multimodal-ocr.md |
| Extract tables and forms | docs/06-layout-structure-and-kie.md |
| Compare all models quickly | comparisons/ |
| Pick a cloud API | models/cloud-apis/ |
| Process invoices | use-cases/invoice-extraction.md |
| Deploy to production | deployment/ |
| Evaluate OCR quality | benchmarks-and-evaluation/ |
| Feed OCR into an LLM pipeline | use-cases/ocr-to-rag.md |
| See model rankings | benchmarks-and-evaluation/leaderboards/ |
Accuracy on hard documents
▲
High │ ┌──────────────────────────────────────────────┐
│ │ VLM-based (OCR-specialised) │
│ │ GOT-OCR · Nanonets · Mistral OCR │
│ │ OlmOCR · DeepSeek-OCR · PaddleOCR-VL │
│ ├──────────────────────────────────────────────┤
│ │ General-purpose VLMs │
│ │ Claude · GPT-4o · Gemini · Qwen-VL │
│ ├──────────────────────────────────────────────┤
│ │ Modern deep learning │
│ │ Surya · PaddleOCR · TrOCR · DocTR │
│ ├──────────────────────────────────────────────┤
│ │ Classic deep learning │
│ │ EasyOCR · Tesseract v5 · Kraken │
│ ├──────────────────────────────────────────────┤
Low │ │ Classical / pre-DL │
│ │ Tesseract v3 · OCRAD · GOCR │
│ └──────────────────────────────────────────────┘
└──────────────────────────────────────────────────►
Low cost / fast High cost / slow
docs/ — Concepts & Learning Path
- History of OCR
- How OCR Works
- OCR Taxonomy
- Document vs Scene OCR
- VLMs and Multimodal OCR
- Layout, Structure & KIE
- Handwriting OCR
- Multilingual OCR
- Benchmarks & Metrics
- Costs, Latency & Scaling
- Failure Modes & Evaluation
- Future of OCR
models/ — Model Reference
- Pre-deep learning — Classical, statistical, commercial
- Deep learning — CNN-LSTM-CTC, attention, transformer, end-to-end
- Scene text — Detection, recognition, end-to-end
- VLM-based — General-purpose and OCR-specialised VLMs
- Cloud APIs — AWS, Azure, Google, OpenAI, Anthropic, Mistral, ABBYY, Mathpix, Baidu, Apple
benchmarks-and-evaluation/ — Measuring Quality
- Metrics — CER, WER, ANLS, TEDS, F1, Edit Distance
- Document benchmarks
- Scene text benchmarks
- VLM benchmarks
- Leaderboards
use-cases/ — Implementation Guides
- Document OCR
- Invoice Extraction
- ID Document OCR
- Scene Text & Mobile
- OCR to RAG
- Hybrid & Ensemble OCR
comparisons/ — Decision Guides
- Best for Documents
- Best for Scenes
- Best for Handwriting
- Best for Multilingual
- Best Low Cost
- Open Source vs API
deployment/ — Production Patterns
- Self-Hosting — Docker, Kubernetes, queues
- Cloud Deployment — AWS, Azure, GCP
- Edge & Mobile — iOS, Android, embedded
- Batch vs Streaming
- Monitoring & Quality
resources/ — Community & Research
See CONTRIBUTING.md. PRs welcome for new models, benchmarks, use cases, and corrections.