Open benchmark for burned-in video subtitle OCR — PP-OCRv6 small beats medium here, and watermark interference is a universal blind spot

[YeefyisOK]

We built a benchmark for a problem general OCR benchmarks don't cover: burned-in video subtitles that sit next to watermarks/credits. 600 samples, 6 languages, ~19% with a synthetic overlapping watermark to test detection robustness specifically.

Tested raw (no post-filtering) detection+recognition across your PP-OCRv6 tiny/small/medium, PP-OCRv5 (our own bundled ONNX build), EasyOCR, and Tesseract. A few findings that might be useful to you:

1. PP-OCRv6 small beats medium on this set — 0.633 vs 0.629 CER (basically tied) but 3.7x faster (381ms vs 1399ms on Apple Silicon CPU). Medium doesn't buy anything here.
2. Watermark-adjacent text tanks every engine's CER by 2-2.6x, PP-OCRv6 included — 0.49 → 1.25 CER when a second text region overlaps the subtitle. Seems like a shared blind spot across detection-based OCR generally, not specific to any one model.
3. On Apple Silicon specifically, we noticed paddlepaddle has no GPU/Metal backend (checked via paddle.device.is_compiled_with_mps — doesn't exist) — every tier is CPU-bound on Mac regardless of size. Might be worth an issue on its own if that's not already tracked.

Repo + eval script: https://github.com/GeekLinkDev/geeklink-ocr-benchmark

Disclosure: I'm the developer of GeekLink, which uses PP-OCR models for subtitle extraction.

[YeefyisOK]

Quick correction: found a real bug after posting this — 437 samples (38%) had ground-truth text wider than the video frame, so the rendered subtitle got clipped off-screen. Fixed with proper line wrapping and extended the set from 600 to 1,140 samples in the same pass. Numbers moved slightly (PP-OCRv6 medium/small are now statistically tied rather than small edging ahead, watermark penalty is a bit larger at ~2.6-2.9x). Repo and README both updated: https://github.com/GeekLinkDev/geeklink-ocr-benchmark