2.2.27 Backend Chandra

Chandra 2

Handle: chandra
URL: http://localhost:35080

Chandra 2 is Datalab's OCR model: it turns documents and images into structured Markdown, HTML or JSON, keeping tables, forms, math and reading order intact. Harbor serves it with vLLM behind an OpenAI-compatible API, so any client that speaks /v1/chat/completions with image content can use it.

Key Features:

• Layout-aware OCR: tables, forms, handwriting, math and multi-column reading order
• Structured output: Markdown, HTML or JSON, not a flat text dump
• OpenAI-compatible: /v1/chat/completions with image_url content parts
• Served as chandra: the OpenAI model name is chandra regardless of the checkpoint
• Shared model cache: weights land in the standard Hugging Face hub layout used by other Harbor services
• Official CLI: the upstream chandra client can point at this server via VLLM_API_BASE

Starting

Chandra needs an NVIDIA GPU. The base compose file reserves no devices, so the nvidia cross-file must be part of the selection:

# Pull the vLLM runtime image (~10GB)
harbor pull chandra

# Start Chandra with GPU passthrough
harbor up chandra nvidia

• The first start downloads the model weights (~10GB) into the Hugging Face cache; the container only reports healthy once vLLM answers /health. The healthcheck allows a 30 minute grace period for that download
• datalab-to/chandra-ocr-2 is not gated — no HARBOR_HF_TOKEN is needed to download it
• Starting plain harbor up chandra (without nvidia) gives the container no GPU and vLLM fails to initialise — always include nvidia
• There is no ROCm variant of this service. AMD hosts are untested: vllm/vllm-openai is a CUDA image, and a ROCm setup would need a different image (rocm/vllm) plus /dev/kfd + /dev/dri passthrough. Do not add nvidia on an AMD host — Docker fails with could not select device driver "nvidia" with capabilities: [[gpu]] before the container starts

Quick check from the host:

curl http://localhost:35080/v1/models

OCR an image (any OpenAI-compatible client works; the model name is always chandra):

IMG=$(base64 -w0 ./page.png)
curl http://localhost:35080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "chandra",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,'"$IMG"'"}},
        {"type": "text", "text": "Convert this page to markdown."}
      ]
    }],
    "max_tokens": 4096
  }'

Using the official chandra CLI against the Harbor server:

pip install chandra-ocr
VLLM_API_BASE=http://localhost:35080/v1 chandra ./page.pdf --method vllm --output_dir ./out

Configuration

Environment Variables

Following options can be set via harbor config:

# Host port the OpenAI-compatible API is exposed on
HARBOR_CHANDRA_HOST_PORT      35080

# vLLM runtime image and tag
HARBOR_CHANDRA_IMAGE          vllm/vllm-openai
HARBOR_CHANDRA_VERSION        v0.17.0

# Hugging Face model id served under the OpenAI model name "chandra"
HARBOR_CHANDRA_MODEL          datalab-to/chandra-ocr-2

# Extra vLLM CLI flags, appended verbatim after Harbor's own flags
# e.g. "--tensor-parallel-size 2 --max-model-len 8192"
HARBOR_CHANDRA_EXTRA_ARGS

# Shared with other services: Hugging Face cache directory and token
HARBOR_HF_CACHE
HARBOR_HF_TOKEN

Harbor launches vLLM with the flags from upstream's chandra_vllm launcher:

--model $HARBOR_CHANDRA_MODEL --served-model-name chandra --dtype bfloat16
--max-model-len 18000 --gpu-memory-utilization 0.85 --enable-prefix-caching
--mm-processor-kwargs '{"min_pixels":3136,"max_pixels":6291456}'
--max-num-seqs 16 --max-num-batched-tokens 2048

--mm-processor-kwargs bounds the pixel budget the vision tower sees; those are the values Chandra was tuned for, not vLLM's defaults. --max-num-seqs 16 is not a tuning knob — at vLLM's default of 256 the CUDA graph warm-up trips an assertion in the hybrid Qwen3.5/GDN attention path and the engine never starts (see Troubleshooting); --max-num-batched-tokens 2048 matches upstream's low-VRAM tier. Anything you put in HARBOR_CHANDRA_EXTRA_ARGS is appended after these flags, and the later occurrence wins in vLLM's argument parser, so you can still raise them or lower --gpu-memory-utilization / --max-model-len if the model does not fit.

harbor config set chandra.extra_args "--gpu-memory-utilization 0.7"
harbor up chandra nvidia

GPU

compose.x.chandra.nvidia.yml reserves all NVIDIA GPUs for the container:

deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]

The base compose.chandra.yml declares no device reservation, which is why nvidia is mandatory in the selection.

16GB of VRAM is enough for the full --max-model-len 18000 context: on an RTX 4090 Laptop (16GB) the defaults above load 8.6 GiB of weights, leave ~3.9 GiB of KV cache (~5.9x concurrency at 18000 tokens) and sit at ~13.9GB steady — idle and under load, since vLLM preallocates at --gpu-memory-utilization 0.85. Smaller cards need --max-model-len lowered through HARBOR_CHANDRA_EXTRA_ARGS.

Volumes

• ${HARBOR_HF_CACHE} (default ~/.cache/huggingface, falling back to ./services/chandra/.hf-cache) is mounted at /root/.cache/huggingface. Weights land in the standard models--<org>--<name> layout, shared with vLLM, TEI and other Harbor services, so a model pulled once is never downloaded twice
• services/chandra/override.env holds per-service environment overrides (harbor env chandra)

License

The Chandra 2 weights are released under the OpenRAIL-M license: free for research and commercial use, but subject to the use restrictions listed in the license. Review them before deploying it in a product. The vLLM runtime itself is Apache-2.0.

Integration with Harbor

Chandra exposes a plain OpenAI-compatible endpoint, so any Harbor service that accepts a custom base URL can use it:

From	Setting
Inside the Harbor network	http://chandra:8000/v1, model chandra
From the host	http://localhost:35080/v1, model chandra
The upstream CLI	VLLM_API_BASE=http://localhost:35080/v1 chandra <file> --method vllm

Any API key value is accepted. Because the served model name is fixed to chandra, swapping HARBOR_CHANDRA_MODEL to another checkpoint does not require reconfiguring the clients.

Chandra is a vision model without general chat ability — point document/OCR tools at it, and keep a separate backend (llama.cpp, Ollama, vLLM) for chat.

Troubleshooting

Check Logs

harbor logs chandra

• harbor logs chandra follows the log; use docker logs harbor.chandra for a one-shot dump
• The first start is dominated by the weight download; the log shows the vLLM engine init only after it finishes

Container never becomes healthy

• could not select device driver "nvidia" means the NVIDIA container toolkit is missing, or the host has no NVIDIA GPU — see the note about ROCm above
• assert num_cache_lines >= batch / AssertionError inside causal_conv1d_update during compile_or_warm_up_model, followed by RuntimeError: Engine core initialization failed. — this is not an OOM. vLLM captures CUDA graphs for batch sizes up to max_num_seqs, and the hybrid Qwen3.5/GDN conv-state cache is smaller than that at the default 256. Harbor already passes --max-num-seqs 16; if you overrode it through HARBOR_CHANDRA_EXTRA_ARGS, put it back. On tight cards --max-num-seqs 2 --max-model-len 8192 also starts cleanly
• No available memory for the cache blocks / CUDA OOM: lower --gpu-memory-utilization or --max-model-len via HARBOR_CHANDRA_EXTRA_ARGS
• Gated or private checkpoints return 401/403 on download until HARBOR_HF_TOKEN is set

Requests fail or the output is truncated

• The model is served as chandra; sending the Hugging Face id as model returns a 404 from the OpenAI API layer
• The raw API response is Chandra's own dialect, not clean Markdown: it can start with an empty <think></think> block and carries HTML-ish markup, including unbalanced tags such as a stray </div>. The upstream chandra CLI post-processes this; a client talking to /v1/chat/completions directly has to strip it
• Very large scans can exceed the --max-model-len 18000 context together with the requested output; downscale the image or raise the limit in HARBOR_CHANDRA_EXTRA_ARGS
• Raise max_tokens in the request for long pages — vLLM stops at the requested budget without an error

Links

• Chandra GitHub Repository
• Chandra 2 on Hugging Face
• Datalab
• vLLM Documentation