[Document Understanding] Can we support a new task on document understanding?

[jlia0]

Document Understanding

Some example models:

1. DiT: https://huggingface.co/microsoft/dit-large
2. LayoutLMv3: https://huggingface.co/microsoft/layoutlmv3-large
3. Donut: https://huggingface.co/docs/transformers/model_doc/donut

Reason for request

Document understanding is a very popular task which I couldn't find any supports for the web environment.

Some tasks include:

1. Key Information Extraction (KIE)
2. Document Layout Analysis (DLA)
3. Document Question Answering (DQA)
4. Optical Character Recognition (OCR)

[xenova]

Those do sound like quite interesting use-cases! Do you mind sharing example code for how you would use the models, as well as the inputs and expected outputs?

[jlia0]

Here's an example code using detectron2 and DiT on document layout analysis.

DiT Doc: https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/dit
HF Space: https://huggingface.co/spaces/imjliao/dit-document-layout-analysis/blob/main/app.py

[xenova]

The repo you shared is private, but I assume I can use this one: https://huggingface.co/spaces/nielsr/dit-document-layout-analysis

[jlia0]

The repo you shared is private, but I assume I can use this one: https://huggingface.co/spaces/nielsr/dit-document-layout-analysis

Oh yes sorry! I forgot it's my private repo. But you're correct, I am using that one as well.

How do you think we can include this to transformer.js? Seems like there is a dependency issue of detectron2...

[xenova]

Hmm, that might complicate things somewhat... Perhaps there is a JS library out there which is a suitable substitute?

[jlia0]

Hmm, that might complicate things somewhat... Perhaps there is a JS library out there which is a suitable substitute?

I don't see a JS library out there could do similar stuffs. But I found something that's worth checking out:

https://github.com/Unstructured-IO/unstructured-inference/blob/main/unstructured_inference/models/detectron2onnx.py

^^^ This is a working example of detectron2 using ONNXRuntime...

[xenova]

Just an update on this:

• Add support for DonutSwin models (Closes #318) #320 added support for Donut models (document parsing and document question answering)
• Add support for TrOCR models (for Optical Character Recognition) #375 will add support for TrOCR models (optical character recognition)

The other tasks (Key Information Extraction and Document Layout Analysis) might be slightly more difficult to add (due the their additional dependencies)... but we'll get there eventually :)

[martinsomm]

Dear @xenova :) I'm trying to implement https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.DocumentQuestionAnsweringPipeline while importing transformers.js from cdn as described in https://huggingface.co/docs/transformers.js/main/en/tutorials/vanilla-js#step-2-javascript-setup.

I get error below

transformers:187 Uncaught Error: This pipeline is not yet supported in Transformers.js v3. at Function._call (transformers:187:16286) at e (transformers:214:134)

Wondering how to circumvent this issue..

Thanks and best regards
Martin

[xenova]

Thanks @martinsomm for the report - this will be fixed by #987.

[martinsomm]

Hi @xenova and wow, thanks for the swift turnaround, very much appreciate your effort :) now I'm wondering when the change will be available on jsdelivr as I still get the same exception, even after "empty cache and hard refresh" in browser.

Best regards, Martin

[xenova]

@martinsomm We've now published https://www.npmjs.com/package/@huggingface/transformers/v/3.0.1, so you can import it from jsdelivr using:

import { pipeline } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.0.1';

[martinsomm]

@xenova thanks again :) now I get updated version from jsdelivr.. this is probably the wrong channel to address another question.. I was wondering what it would take to make batch processing possible, so we can ask multiple questions per image. I can see a comment that currently only batch size of 1 is supported for the DocumentQuestionAnsweringPipeline