DocumentQuestionAnswering widget

[mishig25]

DocumentQuestionAnswering by @ankrgyl

**first of all, amazing work @ankrgyl !

I'm opening a PR here from @ankrgyl branch.

Demo

https://63243249748e56079f18d750--huggingface-widgets.netlify.app/impira/layoutlm-document-qa

API shape

input (identical to VisualQuestionAnsweringWidget ):

const requestBody = {
    inputs: { 
        question: string, 
        image: string, // base64 str of img 
    },
};

output (similar to VisualQuestionAnsweringWidget):

Array<{ answer: string; score: number }>

todos:

• test when api-inference dqa is up
• test widget input samples
• document widget input sample

cc: @NielsRogge @osanseviero

Questions:

@ankrgyl @Narsil Is the API shape for pipelines visual-questions-answering & document-questions-answering are identical? If so, we can just reuse VisualQA widget for DocumentQA pipeline as well. Please let me know

[Narsil]

@ankrgyl @Narsil Is the API shape for pipelines visual-questions-answering & document-questions-answering are identical? If so, we can just reuse VisualQA widget for DocumentQA pipeline as well. Please let me know

It is in the default case. There was actually a debate if they should be different or not.
What settle my opinion is that DocumentQA should be able to work on PDF without looking at them like images (but inspecting the content itself directly). This is not implemented yet, but it means that's it's different in Input/Output (PDF are a document not an image, not the same Python object).

But for the widget, becuase it's only using defaults, then yes it's the same input/output.

[mishig25]

Oh I see

@Narsil Does it mean that the widget has to support PDF upload?

[Narsil]

Should be transparent for the widget, no ?

In any case neither the API nor the actual pipeline deal with it right now, but it's something that we could/should do.

[mishig25]

I see, in this case, I will just reuse "VisualQA widget for DocumentQA pipeline" until more features are needed

[mishig25]

@Narsil does api-inference support document-questions-answering atm?

[ankrgyl]

It looks like this has already been merged, but one thing I didn't do before pushing the initial PR is fix the examples to be more relevant to document question/answering. Is it possible still to change those? Happy to send a PR.

[ankrgyl]

			widgetData: [
				{
					text: "What animal is it?",
					src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg",
				},
				{
					text: "Where is it?",
					src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg",
				},
			],

should probably be

https://miro.medium.com/max/787/1*iECQRIiOGTmEFLdWkVIH2g.jpeg

			widgetData: [
				{
					text: "What is the invoice number?",
					src: "https://templates.invoicehome.com/invoice-template-us-neat-750px.png",
				},
				{
					text: "What is the purchase amount??",
					src: "https://miro.medium.com/max/787/1*iECQRIiOGTmEFLdWkVIH2g.jpeg",
				},
			],