Document Layout Detection

Python toolkit for document layout region detection on scanned pages and PDFs. The primary workflow runs DocLayout-YOLO on the DocLayNet benchmark, applies optional deterministic and VLM-based refinement, and reports COCO mAP metrics used on the ICDAR leaderboard.

What this project does

Page image (or rendered PDF page)
  → DocLayout-YOLO inference
  → Deterministic refinement (clamp, nested same-class suppression)
  → Optional VLM refinement (overlap / low-confidence candidates)
  → COCO detections + overlays + metrics

In scope	Out of scope (for now)
DocLayout-YOLO inference	Retrieval / RAG
DocLayNet mAP evaluation	Chunking benchmarks
Deterministic box refinement	Detector training
Optional VLM class correction	Cloud deployment
OCR context for VLM candidates	DOCX / PPTX parsers

Stack

Python · DocLayout-YOLO · OpenCV · PaddleOCR (optional, for VLM OCR context) · PyMuPDF (PDF pages) · pycocotools

Quick start

1. Install

cd RAG-project
python -m venv .venv
.venv\Scripts\activate          # Windows
# source .venv/bin/activate     # Linux / macOS
pip install -r requirements.txt
pip install doclayout-yolo huggingface_hub pycocotools

On Windows, if pycocotools fails: pip install pycocotools-windows.

2. Download model weights

python scripts/download_model_weights.py

Weights default to juliozhao/DocLayout-YOLO-DocLayNet-Docsynth300K_pretrained (see config/default.yaml).

3. Download DocLayNet eval data

python scripts/download_doclaynet.py

This materializes a YOLO-format val split under data/doclaynet/.

4. Run detection evaluation

python scripts/run_doclaynet_eval.py --data-dir data/doclaynet --max-samples 50 --device cuda:0

Use --device cpu if no GPU is available. Omit --max-samples to evaluate the full split.

Console output includes per-image VLM candidate counts:

  image 1 (0.png): vlm_candidates=1
  image 2 (1.png): vlm_candidates=1
  ...

Configuration

Main settings live in config/default.yaml.

Section	Purpose
layout.detector	YOLO weights, confidence/IoU thresholds, image size, device
layout.refinement	Deterministic post-processing and VLM candidate selection
vlm	Optional vision-language refinement (enabled: false captures inputs only)
ocr	PaddleOCR settings used when building VLM candidate text context
doclaynet	Default data directory, split, and sample cap

Detector backend: set layout.backend: yolo (default). Set layout.backend: heuristic for geometry-only layout from OCR lines (no torch).

VLM null step: with layout.refinement.enabled: true and vlm.enabled: false, the pipeline still selects VLM candidates, writes annotated inputs under reports/doclaynet/vlm_inputs/, and skips API calls.

Evaluation outputs

Reports are written to reports/doclaynet/ (cleared on each eval run unless --no-clear).

File / folder	Content
doclaynet_metrics.json	mAP @ IoU 0.50:0.95, AP50, annotation counts
DOCLAYNET_LEADERBOARD.md	Your scores vs published DocLayout-YOLO references
doclaynet_predictions.json	Refined COCO-format detections
doclaynet_detector_predictions.json	Raw detector output (before refinement)
doclaynet_confusion.json	Matched GT vs predicted class pairs (IoU ≥ 0.5)
doclaynet_vlm_candidates.json	Per-image slim VLM candidate records
doclaynet_vlm_class_changes.json	Class flips / removals applied by VLM
doclaynet_vlm_errors.json	Per-image refinement failures
overlays/	Detector boxes drawn on each page
overlays_vlm_refinement/	Boxes after refinement
vlm_inputs/images/	Full-page images with all boxes; candidates labelled
vlm_inputs/prompts/	System + user prompts for each VLM call

VLM candidate JSON shape

Each image entry in doclaynet_vlm_candidates.json looks like:

{
  "image_id": 17,
  "filename": "16.png",
  "candidate_count": 2,
  "vlm_candidates": [
    {
      "box_id": "b17",
      "class_name": "Section-header",
      "vertical_position": "top_margin",
      "horizontal_position": "left",
      "ocr_text": "ABC BANCORP SUBSIDIARY BANKS",
      "overlaps_with": ["b13"]
    }
  ]
}

Detection pipeline (detail)

1. Inference — DocLayout-YOLO predicts DocLayNet classes (Caption, Text, Table, Picture, …) with bounding boxes.
2. Deterministic refinement — clamp boxes to page bounds; remove nested same-class duplicates above a containment threshold.
3. VLM candidate selection — flag boxes that overlap another region (IoU ≥ vlm_cluster_iou_threshold) or fall below vlm_low_confidence_threshold; expand overlap partners transitively.
4. Optional VLM call — send full-page image + candidate metadata; apply allowed class flips and overlap-gated removals.
5. Metrics — convert predictions to COCO format and compute mAP with pycocotools.

Reference scores (DocLayout-YOLO, Docsynth-pretrained on DocLayNet): mAP 79.7%, AP50 93.4%.

PDF detection (secondary)

You can run layout detection on local PDFs (render → detect → optional refinement):

python scripts/run_pipeline.py data/pdfs/sample.pdf --viz

This writes structured output under outputs/ and optional overlay PNGs. The PDF path shares the same detector and refinement modules as the DocLayNet eval; it is useful for spot-checking boxes on real documents.

Project layout

RAG-project/
├── ARCHITECTURE.md           # module boundaries and artifact contracts
├── config/default.yaml
├── data/
│   ├── doclaynet/            # eval images + YOLO labels
│   └── pdfs/                 # optional local PDFs
├── scripts/
│   ├── download_doclaynet.py
│   ├── download_model_weights.py
│   ├── run_doclaynet_eval.py
│   ├── build_doclaynet_ideal_boxes.py
│   └── run_pipeline.py       # PDF render + detect (optional)
├── src/layout_pdf_rag/       # core package
└── reports/doclaynet/        # eval metrics, overlays, VLM inputs

See ARCHITECTURE.md for module responsibilities, data contracts, and extension rules.

Other scripts

Script	Purpose
build_doclaynet_ideal_boxes.py	Render ground-truth boxes onto DocLayNet images
download_doclaynet.py	Fetch labelled val/train subset from HuggingFace
download_model_weights.py	Cache DocLayout-YOLO checkpoint locally

License

MIT (add your own license if needed)