ViTLP

This repository releases code of paper Visually Guided Generative Text-Layout Pre-training for Document Intelligence (NAACL-2024).

We provide the pre-trained checkpoint ViTLP-medium (380M). The pre-trained ViTLP model can natively perform OCR text localization and recognition, which is accessible at Huggingface. Clone (or download) ViTLP checkpoint weight to the directory ./ckpts/ViTLP-medium.

pip install -r requirements.txt
git clone ViTLP && cd ViTLP

# Clone ViTLP-medium checkpoint
mkdir ckpts
git clone https://huggingface.co/veason/ViTLP-medium ckpts

Demo

With the checkpoint and dependencies set (see requirements.txt), run the demo as

python ocr.py

Upload a document image and have a shot

See detailed inference code at decode.py and run batch decode by

bash decode.sh

Preset FAQ

• Why is ViTLP-medium (380M)?

  When I commenced this project, it was on the eve of LLMs (precisely speaking, ChatGPT). ViTLP-base presented in our paper, is actually a rather small pre-trained model. We know it is expected to scale up ViTLP in this LLM era. However, the pre-training scale is commonly constrained by computation resources and the pre-training dataset scale, in which context ViTLP-medium (380M) is the largest pre-training scale so far we can support.

  Besides, this scale of ViTLP also brings inference sweetness including speed and memory usage. Typically, OCR on a page of a document image can be processed within 5~10 seconds in an Nvidia 4090, which is comparable to (and faster than) most OCR engines (and LLMs).

Note

ViTLP is pronounced /ˈvai·tlp/ (vital). The first version of our paper was submitted to OpenReview in June 2023.