Feature request: support for local AI models

[lordekeen]

Hi, would it be possible to include support for local AI models that are OCR capable through OpenAI Compatible API? This way the user can configure its own llm server (like llama.cpp), there are some pretty capable local models that can run in consumer hardware. That would be great for privacy as the documents don't need to leave the user's machine.

[jritzi]

Great idea! This is actually possible today using the custom command option, but that requires writing a script to get things working. I agree that it would be good to have an OpenAI compatible API option to make it much more straightforward to run local models. I'll add that to my list of features to implement in the future.

[jritzi]

If you want a workaround in the meantime, here's a (Claude-generated) Python script that should be a good starting point. For example:

1. Install Ollama
2. Run ollama pull glm-ocr
3. In the OCR Extractor plugin settings, choose "Custom command"
4. As the command, use python3 /path/to/script.py {input} {output}
5. Enable "Convert PDFs to images"

#!/usr/bin/env python3
import sys, base64, json
from pathlib import Path
from urllib.request import Request, urlopen

input_path, output_path = sys.argv[1], sys.argv[2]

MIME_TYPES = {".png": "image/png", ".jpg": "image/jpeg", ".jpeg": "image/jpeg", ".webp": "image/webp"}
mime = MIME_TYPES.get(Path(input_path).suffix.lower(), "image/png")

data_url = f"data:{mime};base64,{base64.b64encode(Path(input_path).read_bytes()).decode()}"

payload = json.dumps({
    "model": "glm-ocr",
    "messages": [
        {
            "role": "system",
            "content": "You are an OCR extractor. Extract all text from the document page in the image. Output the text as markdown, preserving the document structure including headings, lists, and tables. Output only the extracted text — no commentary, explanations, or surrounding prose."
        },
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": data_url, "detail": "high"}}
            ]
        }
    ],
    "temperature": 0,
    "max_tokens": 4096
}).encode()

req = Request(
    "http://localhost:11434/v1/chat/completions",
    data=payload,
    headers={"Content-Type": "application/json"}
)
result = json.loads(urlopen(req).read())
Path(output_path).write_text(result["choices"][0]["message"]["content"])

[lordekeen]

Nice, i'll see if i can test this later. I already have a working llama.cpp instance just need to find a good OCR model.

[jritzi]

I just released a new version that has an "OpenAI-compatible API" option to support this natively without needing a custom script. Feel free to try it out and let me know how it goes.

[lordekeen]

I've just tested with PaddleOCR-VL-1.6 running on llama.cpp and it worked very well (and fast):

I've had some issues understanding tables structure, but thats probably on the model and not your implementation:

Trying to extract text from screenshots makes the model hallucinate, i'll look for other models.

[lordekeen]

Testing with Qwen3.5 4B in non-thinking mode the output inside the llama.cpp web-ui is perfect. even with the tables:

Inside Obsidian it fails if the image is in .webp:

If the image is in .jpg or .png, the output is correct:

PaddleOCR succeeds in extracting the text, but fails to reconstruct the tables. Qwen is the winner in this case.

[jritzi]

Thanks for all the testing details, very helpful! Looking into the issue with WebP you ran into, it looks like llama.cpp doesn't support that format. The plugin already handles resizing images, converting PDF pages to images, etc., so I should be able to fix this by converting the image if it isn't already in a supported format.

[lordekeen]

I should thank you, this is working really well! Here are my llama.cpp settings in case you or anyone reading this want to try:

This is the unsloth/Qwen3.5-4B-UD-Q4_K_XL.gguf, notice i use split layer and manual fit as its better for my dual gpu setup, but normally you don't need to declare fit or split mode.

n-gpu-layers = auto
fit = off
ctx-size = 32768
split-mode = layer
tensor-split = 1,0
cache-type-k = q8_0
cache-type-v = q5_1
cache-type-k-draft = q8_0
cache-type-v-draft = q5_1
spec-type = draft-mtp
spec-draft-n-max = 2
spec-draft-p-min = 0.9
temp = 0.7
top-p = 0.8
top-k = 20
min-p = 0
presence-penalty = 1.5
repeat-penalty = 1.0
chat-template-kwargs = {"enable_thinking":false}
reasoning = off

This has given me the best results, extracting tables perfectly, it uses around 6gb of vram. I've also tried the Qwen3.5 2B to save more vram, but it struggles more with text structure, the 4B one was the sweet spot. Dunno if 32k context is necessary, reducing it might make the model fit in under 4gb vram.

[jritzi]

Just a note that I released version 2.3.2 which should fix the issue with WebP images not working.