Multi GPU Support

[akhiljr10]

Question

from docling.datamodel.base_models import InputFormat
from docling.document_extractor import DocumentExtractor, ExtractionFormatOption
from docling.datamodel.accelerator_options import AcceleratorOptions, AcceleratorDevice
from docling.datamodel.pipeline_options import VlmExtractionPipelineOptions
from docling.backend.pypdfium2_backend import PyPdfiumDocumentBackend
from docling.pipeline.extraction_vlm_pipeline import ExtractionVlmPipeline

pdf_options = AcceleratorOptions(num_threads=16, device=AcceleratorDevice.CUDA)

pdf_opt = ExtractionFormatOption(
    pipeline_cls=ExtractionVlmPipeline,
    pipeline_options=VlmExtractionPipelineOptions(accelerator_options=pdf_options),
    backend=PyPdfiumDocumentBackend
)

extractor = DocumentExtractor(
    allowed_formats=[InputFormat.PDF],
    extraction_format_options={InputFormat.PDF: pdf_opt}
)

I'm running this on virtual machine with 16 vCPUs, 104 GB RAM and 2 NVIDIA T4, and still running into error

Error processing page 1: CUDA out of memory. Tried to allocate 6.00 GiB. GPU 0 has a total capacity of 14.58 GiB of which 5.65 GiB is free. Including non-PyTorch memory, this process has 8.93 GiB memory in use. Of the allocated memory 8.73 GiB is allocated by PyTorch, and 74.44 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Checking nvidia-smi shows that only one GPU is being used.

any suggestions or thoughts!

[dosubot]

Docling currently uses only one GPU per process and does not support multi-GPU parallelism out of the box (no DataParallel/DistributedDataParallel). You can manually utilize both GPUs by running separate Docling processes, each with device='cuda:0' or device='cuda:1' in your AcceleratorOptions—but a single extraction job won't span multiple GPUs automatically [source].

To address the CUDA out of memory error, try reducing batch size, processing smaller page ranges, lowering model scale, reducing input resolution, or disabling expensive pipeline features. Setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True can help reduce memory fragmentation [source]. Explicitly deleting models/tensors and calling torch.cuda.empty_cache() after processing can also help free up GPU memory [source].

For high-throughput scenarios, consider distributed processing with docling-serve, which lets you parallelize workloads across devices [source]. If you need true multi-GPU support for a single job, architectural changes would be required. Let me know if you want more details on any workaround!

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.