GLM-OCR: how to use it

[mrsipan]

llama.cpp has added support for glm-ocr:

#19677

But currently there is not examples of how to use it. Can someone provide examples of it?

[SmartestWashingMachine]

1. Download a decoder & mmproj gguf via Huggingface
2. Run latest llama.cpp server binary with command llama-server.exe -m glmocr-Q4_K_M.gguf --mmproj mmproj-glmocr-Q4_1.gguf -c 12000 -ngl 99 --flash-attn off -fit off (replace the -m and --mmproj paths with the GGUFs you downloaded)

(seems like flash-attn must be off for it to work currently.)

3. in a Python script, call the llama-server GLM-OCR model like so:

import base64
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="blahblah-probablynotneeded")

def encode_image(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

# Path to your local image
image_path = "./some_image.png"
base64_image = encode_image(image_path)

response = client.chat.completions.create(
    model="model",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{base64_image}"}
                },
                {"type": "text", "text": "Text Recognition:"},
            ],
        }
    ],
    temperature=0.02,
)

print(response.choices[0].message.content)

[xms991]

I have a very basic implementation working, and I've provided my config.ini and python script below for reference.

I'm trying to figure out how to get more functionality out of it, and I'd really appreciate any guidance you can provide. Specifically I am trying to determine:

1. How can I control things like detection modes and output modes?

2. How can I utilize this within a more full-featured layout detection pipeline, and ideally some kind of UI for rendering detected layout features?

3. I see the GLM team provides a guide on using Ollama for local deployment (see: https://github.com/zai-org/GLM-OCR/blob/main/examples/ollama-deploy/README.md), but I don't want to use Ollama unless absolutely necessary. It seems like they may be using a custom endpoint instead of a normal v1/chat/completions endpoint.

config.ini for llama-server:

[GLM-OCR-f16]
LLAMA_ARG_CACHE_TYPE_K = f16
LLAMA_ARG_CACHE_TYPE_V = f16
mmproj = /models/mmproj-GLM-OCR-Q8_0.gguf
c = 131072
ngl = 99
flash-attn = off
fit = off

Python script:

import base64
import requests
import pymupdf

url="http://my-server-name.local:8080/v1/chat/completions"

pdf_path = "Payslip_to_Print_-_Report_Design_01_20_2026.pdf"

def pdf_to_b64_pngs(pdf_path):
    doc = pymupdf.open(pdf_path)
    b64_images = []
    
    for page in doc:
        pix = page.get_pixmap()
        
        png_bytes = pix.tobytes("png")
        
        b64_string = base64.b64encode(png_bytes).decode('utf-8')
        b64_images.append(b64_string)
        
    doc.close()
    return b64_images

def scan_pdf(pdf_path):
    
    b64_images = pdf_to_b64_pngs(pdf_path)

    headers = {"accept":"application/json"}

    responses = []
    
    for b64_image in b64_images:
        payload = {"model": "GLM-OCR-f16",
                "messages": [{
                    "role": "user",
                    "content": [
                        {
                            "type": "image_url",
                            "image_url": {"url": f"data:image/png;base64,{b64_image}"}
                            },
                            {
                                "type": "text", "text": "Text Recognition:"
                                },
                                ],
                            }
                            ],
            "temperature": 0.02}


        response = requests.post(url=url, headers=headers, json=payload).json()

        responses.append(response)

    return responses
    

responses = scan_pdf(pdf_path)

for response in responses:
    print(response['choices'][0]['message']['content'])

[SmartestWashingMachine]

@xms991

No worries, GLM-OCR is rather confusingly documented. To the best of my understanding:

1. GLM-OCR states the list of modes here

So for the formula recognition mode as an example, we would just replace the content of the text field in that code snippet of yours, from Text Recognition: to Formula Recognition:

Note that GLM-OCR does NOT support detection by itself. It can only recognize text. LLMs tend to suffer when it comes to predicting integer coordinates as tokens. This ties into point 2...

2. re: text detection - GLM-OCR assumes we use a separate model pipeline to detect text regions. The developers used PP-DocLayoutV3 which required the PaddlePaddle library. But it should be fairly usable with just about any detection pipeline.

If we want to use PaddlePaddle for layout/text detection we can adapt the example given here:

from paddlex import create_model

model_name = "PP-DocLayout_plus-L" # Note this is NOT the model they used, this is just a quick example.

model = create_model(model_name=model_name)
output = model.predict("layout.jpg", batch_size=1, layout_nms=True)

for res in output:
    # I don't remember the exact "res" interface from the top of my head, but the goal is to crop the image using this detected result and pass it to GLM OCR. Alternatively, we can save_to_img() below and then just use that image path (though this is rather inefficient).
    res.save_to_img(save_path="./output/")
    res.save_to_json(save_path="./output/res.json")

That's just one variant - Paddle has many others in their docs. Apparently Huggingface transformers (pytorch) now supports this too?

If dependencies are a big concern, an ONNX variant of the PP models can be used e.g: this - but this implementation was buggy if I remember right.

2. re: UI - Unfortunately there's no ready-made UI builder for this task. As cliche as it sounds, prompting an LLM (even plain-old Gemini) to vibe code the UI/Simple Backend is probably the best bet here. The overall approach is something like:

User makes "OCR request" (whether through your Python script or this frontend UI) -> Backend uses text detection pipeline to detect text regions (usually PaddlePaddle) -> Feed detected text regions into GLM-OCR through llama.cpp server -> Return detected text features and their corresponding OCR'd contents -> Frontend UI renders it.

3. I can't speak for Ollama as I don't use it, but a glance at their docs seems to imply that this generate route is used due to some weird quirk in Ollama's chat completion route for vision? If you're going for the approach mentioned above this shouldn't be anything to worry about. Probably.

[MartinEls]

There SDK/cli tool is really good in terms of out-of-the-box performance for pdf2MD without parameter fiddling. If you want to feed a pdf into LLMs, it covers many relevant document classes. On a M-Series MAC, here is what worked for me:

I downloaded the GGUFs from here ggml-org/GLM-OCR-GGUF (the Q8 gguf and the mmproj) and run them with:

# brew install llama.cpp
llama-server -m models/GLM-OCR-Q8_0.gguf --mmproj models/mmproj-GLM-OCR-Q8_0.gguf -c 12000 -ngl 99 --flash-attn off -fit off

The model is now useable from the GLM OCR cli pipeline (this includes a second, smaller model for segmentation, these segments get then feed to the OCR model running with llama.cpp).

# pip install glmocr
time glmocr parse testdocs/RFeynman_plentySpace.pdf --config config.yaml 

Take the config from the SDK repo and set mass: enabled to false . The default ports should work, adjust if needed.

Needs a bit less than 3 mins for a 7 page text heavy pdf, for denser scientific papers like this or this expect 5 to 10 mins on a M1 MacBook Air (imho not too bad for the quality and a 5y old Ultrabook).

[xms991]

Thanks for sharing! I made some progress and I have it working in basic form via the official GLM SDK/pipeline.

The critical piece of information I was missing was that I could just use a /v1/chat/completions endpoint, which is the default approach utilized by the sglang/vllm implementations. I haven't used sglang or vllm before, so didn't realize until I dug into their config.yaml further.

I'm still trying to troubleshoot some quality issues that I'm experiencing, but my initial confusion about Ollama and endpoints should be resolved now. I am a bit curious why the Ollama implementation uses a different endpoint instead of using a v1/chat/completions endpoint, but not super concerned since I refuse to use Ollama.

[MartinEls]

See some minor issues with reading order and a bit inconsistent inclusion/exclusion of headers/footers that I'm not 100% happy with. With other OCR models I had various problems with repeating patterns or hallucinations in tables; for my test cases I didn't find that behaviour with the GLM OCR model. What I really like here is the cost/quality trade-off, it's on a sweet spot for my use case.

[xms991]

@MartinEls - Would you mind trying to run the model on the first page of the PDF linked below and let me know how the results are? This is just a random example I found via google, and I'm getting really bad results.

Particularly, I am attempting to OCR the states and IDs in the table, which went VERY poorly, but other information such as the address at the top was also significantly off for me.

https://www.content.shi.com/cms-content/accelerator/media/terms/shi-multijurisdictional-resale-form-2023.pdf

A few examples of the types of issues I'm having:

Actual PDF:

OCR output, using GLM-OCR SDK:

My guess was that maybe the PDF conversion and document layout parsing was somehow leading to compression and causing degradation, so I also tested running it directly without the GLM SDK, just feeding a base64 encoded image into the v1/chat/completions endpoint.

Real PDF:

OCR output using v1/chat/completions:

Skipping the layout parsing seems to produce slightly better results in some ways, but they're still insanely bad overall.

Any ideas would be appreciated.

[Disonantemus]

# brew install llama.cpp
llama-server -m models/GLM-OCR-Q8_0.gguf --mmproj models/mmproj-GLM-OCR-Q8_0.gguf -c 12000 -ngl 99 --flash-attn off -fit off

The model is now useable from the GLM OCR cli pipeline (this includes a second, smaller model for segmentation, these segments get then feed to the OCR model running with llama.cpp).

# pip install glmocr
time glmocr parse testdocs/RFeynman_plentySpace.pdf --config config.yaml 

Take the config from the SDK repo and set mass: enabled to false . The default ports should work, adjust if needed.

I've tried that and got stuck using the same:

Previously

• download and edit config.yaml
• glmocr install:

uv tool install glmocr

First run

$ glmocr parse in.jpg --config config.yaml
Loading images: in.jpg
Found 1 file(s)
Error: Layout detector dependencies are missing or incompatible. Try: pip install 'glmocr[layout]'. Original error: ModuleNotFoundError: No module named 'torch'

Reinstall

uv tool install glmocr[layout]

Big download including torch and cuda.

Second Run

$ glmocr parse in.jpg --config config.yaml
Loading images: in.jpg
Found 1 file(s)
Starting Pipeline...
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
preprocessor_config.json: 100%|████████████████████████████████████████████████████████████████████████| 575/575 [00:00<00:00, 4.25MB/s]
config.json: 2.46kB [00:00, 10.9MB/s]
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████| 133M/133M [00:06<00:00, 20.1MB/s]
Loading weights: 100%|██████████████████████████████████████████████████████████████████████████████| 858/85
8 [00:00<00:00, 8270.75it/s]
Pipeline started!
GLM-OCR initialized in self-hosted mode
Using Pipeline (enable_layout=true)...

=== Parsing: in.jpg (1/1) ===
Layout post_process failed for chunk (retrying image-by-image):
PPDocLayoutV3ImageProcessorFast requires the OpenCV library but it was not found in your environment. You can install it with:
```
pip install opencv-python
```
Please note that you may need to restart your runtime after installation.

Layout post_process failed for image 0 in chunk:
PPDocLayoutV3ImageProcessorFast requires the OpenCV library but it was not found in your environment. You can install it with:
```
pip install opencv-python
```
Please note that you may need to restart your runtime after installation.

Stopping Pipeline...
Pipeline stopped!

All done!

Check dependency

$ pacman -Qs opencv
local/opencv 4.13.0-2
    Open Source Computer Vision Library
local/python-opencv 4.13.0-2
    Python bindings for OpenCV

I have it installed, so ... what's the problem?

System

distro: Arch Linux x86_64
kernel: 6.15.2-arch1-1
shell: bash 5.3.9
term: tmux
cpu: Intel i7-4790 (8) @ 3.600GHz
gpu: NVIDIA GeForce GTX 1660 SUPER

[MartinEls]

The table looks ok for me (you inserted 2x the original, so I don't see the issue). The checkbox area above is a bit inconsistent:

Default settings from the cli with the Q8 GGUF as described above. Full output: https://gist.github.com/MartinEls/baffe3873ce2c25e66fb76cdac46f0c5

It seems to ignore the address completely, maybe it makes sense to adjust the confidence scores that trigger the OCR steps for specific components.

[xms991]

Thank you very much for testing!

I updated my previous comment to add the intended image. You can see mine has major issues like repeating state labels and mangled IDs, especially where the ID field contains a mix of text and numbers.

Your output looks nearly perfect, so something is clearly very wrong on my end. I'll try to tinker more and see if I can figure out what's going on.

[xms991]

Did some more testing and it looks like everything is working fine now. It looks like some aspect of my llamacpp server config.ini defaults was the culprit. Not sure exactly which, but using your command with a clean llamacpp install on another computer worked fine. Gutted some stuff from my server config.ini and it is now working fine there too.

Thanks again for helping me test, @MartinEls!