Repository navigation
Implement custom_ocr strategy — MiniCPM-OCR as the reference use case
#17853
lottshin
started this conversation in
Feature Requests & Suggestions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Problem
Uploading a scanned PDF fails with
No text found in document(#17085). OCR is wired to Mistral's/ocrAPI (mistral_ocr,azure_mistral_ocr,vertexai_mistral_ocr) or the built-indocument_parser.OCRStrategy.CUSTOM_OCRis already declared inpackages/data-provider/src/config.ts, butgetStrategyFunctions()inapi/server/services/Files/strategies.jshas no branch for it, so selecting it throwsInvalid file source. That leaves self-hosted and air-gapped deployments (see also #17644) unable to use a local OCR model.Proposal
Complete the
custom_ocrstrategy along the lines of #13431 (a module loaded fromocr.customStrategyModule), with a MiniCPM-OCR module as the reference implementation.MiniCPM-OCR 1.0 is a 1.3B Apache-2.0 OCR model served through an OpenAI-compatible chat-completions endpoint, which makes it a realistic local backend for that reference module.
Why the plugin point rather than a new enum value
The existing strategies name API shapes, not models. A
minicpm_ocrentry would invite one enum value per OCR model, so the pluggable path looks like the right home. #13431 has the design but is currently out of date withdev(it needs a rebase) and has had no maintainer review since June; I am happy to revive it with the author's agreement, or to contribute the reference module on top of it if that is preferred.Open design question: PDF input
Mistral's
/ocraccepts the file directly, while MiniCPM-OCR needs page images. Rendering PDF pages server-side needs a rasterizer —pdfjs-distis already a dependency, but rendering to an image needs a canvas implementation, so this likely means a new dependency or an images-only constraint at first. Guidance on which direction maintainers prefer would be useful before writing code.Scope I can implement
custom_ocrhandler wiring instrategies.js(or the equivalent if 🔌 feat: Pluggable Custom OCR Strategy #13431's shape changes).ocrfields (customStrategyModule);ocr.baseURLandocr.apiKeyalready exist and would be reused. Tests included.POST {ocr.baseURL}/v1/chat/completions, image content part before the prompt, text read fromchoices[0].message.content. Its location can follow whatever the repo prefers — there is noexamples/directory today, so a docs snippet plus a documented module may be a better fit than adding one.document_parserunless that behavior is wanted.librechat.example.yamlsnippet, and documentation.dev, per the contributing guidelines.Related: #13431, #17085, #17644.
All reactions