You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Engine plugins: extraction engines as separate Python packages
#116
content-core has two kinds of engines today, document and URL, and every one of them lives in this repository. This thread is about changing that: letting an engine ship as its own Python package, installed with pip, without a content-core release.
What exists today
Engines are a closed vocabulary. DocumentEngine is auto, simple, docling; UrlEngine is auto, simple, firecrawl, jina, crawl4ai (common/types.py). The names are validated when the config is built, and routing lives in one place (_route_for_mime() in extraction.py).
Heavy engines are optional extras with a guarded import (content-core[docling], content-core[crawl4ai]), following "formats are core, engines are extras" in ARCHITECTURE.md.
So a new engine means a PR here, a review by this project, a new extra, and a release. Every engine also becomes something this project tests and maintains from then on.
That worked while engines were few. Proposals now arrive faster than it makes sense to absorb them: an external Docling Serve API (#45), CRW and fastCRW as URL engines (#79, #85), and most recently MiniCPM-OCR as a document engine for scanned and photographed pages, proposed with a side-by-side comparison in open-notebook#1576.
The idea
A third-party package registers an engine, and content-core discovers it at runtime. Roughly what LangChain does with partner packages:
The plugin author owns the code, the dependencies, the release cadence and the support. content-core owns the contract.
What has to keep holding
These are existing principles in ARCHITECTURE.md, and a plugin system has to respect them rather than work around them:
Explicit choice is a requirement; auto is a preference. A named engine is honored or the call raises. A plugin engine that isn't installed must fail the same way document_engine="docling" does today, with the install command in the message.
Single source of truth for routing.check_file_support must keep telling the truth, including for plugin engines.
The two-layer processor contract and the raise/degrade boundary. A plugin returns an ExtractionOutput or raises a typed error. An empty successful result is an error, not an empty document.
Closed vocabulary validated at the entry point. With plugins the vocabulary becomes "built-ins plus what's installed", and a typo still has to fail at configuration time with the list of valid names.
Questions that decide the design
Discovery. Python entry points (for example a content_core.document_engines group), so installing the package is enough? Or explicit registration by the host application?
What a plugin declares. Name, kind (document or URL), the MIME types it accepts, an availability check, and what else? Does it declare whether it sends content off the machine?
Configuration. Settings today are fields on ContentCoreConfig with the CCORE_ prefix and a fixed precedence. How does a plugin get its own settings (a service URL, a timeout) without content-core knowing its fields in advance?
auto. Does auto ever pick a plugin engine? My starting position is no: plugins are chosen by name only.
Reporting to host applications. Hosts such as Open Notebook need to list which engines are installed and usable, to show them in a settings screen. What does that call look like?
Stability. A plugin contract is a compatibility promise. What is the smallest surface we can commit to, and how is it versioned?
Trust and ownership. A plugin runs arbitrary code on the user's documents. Where do plugins live (their authors' repositories, a list in this README), and what does this project say about ones it doesn't maintain?
Existing extras. Do docling and crawl4ai stay as extras, or move to the same mechanism once it exists?
First candidate
MiniCPM-OCR is the concrete case to design against. The comparison in open-notebook#1576 shows where it helps (photographed receipts, degraded scans, tables with row and column spans) and where Docling stays ahead (text-native PDFs, speed), and it needs a self-hosted service URL, which exercises question 3. Whether content-core adopts that engine is a separate decision from whether plugins exist; with plugins, it wouldn't need to be adopted at all, only installable.
How to help
If you've wanted to add an engine, or you maintain something that could be one, say what your engine needs from the contract: inputs, settings, dependencies, how it fails. A short sketch of the interface you'd want to implement is more useful than agreement in principle.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
content-core has two kinds of engines today, document and URL, and every one of them lives in this repository. This thread is about changing that: letting an engine ship as its own Python package, installed with
pip, without a content-core release.What exists today
DocumentEngineisauto,simple,docling;UrlEngineisauto,simple,firecrawl,jina,crawl4ai(common/types.py). The names are validated when the config is built, and routing lives in one place (_route_for_mime()inextraction.py).content-core[docling],content-core[crawl4ai]), following "formats are core, engines are extras" inARCHITECTURE.md.That worked while engines were few. Proposals now arrive faster than it makes sense to absorb them: an external Docling Serve API (#45), CRW and fastCRW as URL engines (#79, #85), and most recently MiniCPM-OCR as a document engine for scanned and photographed pages, proposed with a side-by-side comparison in open-notebook#1576.
The idea
A third-party package registers an engine, and content-core discovers it at runtime. Roughly what LangChain does with partner packages:
The plugin author owns the code, the dependencies, the release cadence and the support. content-core owns the contract.
What has to keep holding
These are existing principles in
ARCHITECTURE.md, and a plugin system has to respect them rather than work around them:autois a preference. A named engine is honored or the call raises. A plugin engine that isn't installed must fail the same waydocument_engine="docling"does today, with the install command in the message.check_file_supportmust keep telling the truth, including for plugin engines.ExtractionOutputor raises a typed error. An empty successful result is an error, not an empty document.Questions that decide the design
content_core.document_enginesgroup), so installing the package is enough? Or explicit registration by the host application?ContentCoreConfigwith theCCORE_prefix and a fixed precedence. How does a plugin get its own settings (a service URL, a timeout) without content-core knowing its fields in advance?auto. Doesautoever pick a plugin engine? My starting position is no: plugins are chosen by name only.doclingandcrawl4aistay as extras, or move to the same mechanism once it exists?First candidate
MiniCPM-OCR is the concrete case to design against. The comparison in open-notebook#1576 shows where it helps (photographed receipts, degraded scans, tables with row and column spans) and where Docling stays ahead (text-native PDFs, speed), and it needs a self-hosted service URL, which exercises question 3. Whether content-core adopts that engine is a separate decision from whether plugins exist; with plugins, it wouldn't need to be adopted at all, only installable.
How to help
If you've wanted to add an engine, or you maintain something that could be one, say what your engine needs from the contract: inputs, settings, dependencies, how it fails. A short sketch of the interface you'd want to implement is more useful than agreement in principle.
Status: exploring
All reactions