Add LayoutLMv2 OnnxConfig #16309

chainyo · 2022-03-21T19:15:57Z

What does this PR do?

Add LayoutLMv2 OnnxConfig to make this model available for conversion.

I took the same config as LayoutLM and added the adapted shebang.

Who can review?

Models: @LysandreJik @lewtun

HuggingFaceDocBuilderDev · 2022-03-21T19:29:57Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

NielsRogge · 2022-03-22T07:16:11Z

There's quite some interest for ONNX support of LayoutLMv2, see #14368 and #14555.

chainyo · 2022-03-22T08:11:34Z

There's quite some interest for ONNX support of LayoutLMv2, see #14368 and #14555.

Ok thanks for the links, I will look at them when I got some time and start improving the actual PR (it seems that the last PR has been closed due to inactivity, so I will take what I can from this and improve my own PR)

Thanks!

chainyo · 2022-03-23T15:43:27Z

src/transformers/onnx/config.py

@@ -235,6 +235,22 @@ def _generate_dummy_images(
            images.append(Image.fromarray(data.astype("uint8")).convert("RGB"))
        return images

+    def _generate_dummy_bbox(self, batch_size: int = 2, image_height: int = 40, image_width: int = 40) -> List[int]:


I have added a way to generate dummy_bbox if required by the preprocessor. The bounding box are scaled on the image size by default.

chainyo · 2022-03-23T15:44:22Z

src/transformers/onnx/config.py

@@ -245,6 +261,7 @@ def generate_dummy_inputs(
        num_channels: int = 3,
        image_width: int = 40,
        image_height: int = 40,
+        return_bbox: bool = False,


I also added a new arguments to generate_dummy_inputs to handle if the preprocessor needs bounding boxes or not.

chainyo · 2022-03-23T15:45:47Z

src/transformers/onnx/config.py

@@ -295,6 +314,10 @@ def generate_dummy_inputs(
            )
            # Generate dummy inputs according to compute batch and sequence
            dummy_input = [" ".join([preprocessor.unk_token]) * seq_length] * batch_size
+            # Generate dummy bounding boxes if needed by the preprocessor e.g. for LayoutLMv2
+            if return_bbox is True:


If the return_bbox is True then it means the preprocessor needs bounding boxes and it returns the appropriate dictionary with dummy_bbox.

chainyo · 2022-03-23T15:51:56Z

So, with the previous add the dummy inputs generation is working, but it seems that the forward function is facing a problem with the position embedding tensors.

There is a problem when forward is trying to calculate text embeddings, the position_embeddings tensor has not the same shape and can't be added with others

embeddings = inputs_embeds + position_embeddings + spatial_position_embeddings + token_type_embeddings
torch.Size([1, 14, 768]) torch.Size([1, 63, 768]) torch.Size([1, 14, 768]) torch.Size([1, 14, 768])

Here is the error trace :

Traceback (most recent call last):
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/__main__.py", line 99, in <module>
    main()
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/__main__.py", line 81, in main
    onnx_inputs, onnx_outputs = export(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/convert.py", line 308, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/convert.py", line 171, in export_pytorch
    raise err
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/onnx/convert.py", line 148, in export_pytorch
    onnx_export(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/__init__.py", line 275, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 88, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 689, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 458, in _model_to_graph
    graph, params, torch_out, module = _create_jit_graph(model, args,
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 422, in _create_jit_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/onnx/utils.py", line 373, in _trace_and_get_graph_from_model
    torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/jit/_trace.py", line 1160, in _get_trace_graph
    outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
    graph, out = torch._C._create_graph_by_tracing(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
    outs.append(self.inner(*trace_inputs))
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1039, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 893, in forward
    text_layout_emb = self._calc_text_embeddings(
  File "/home/chainyo/miniconda3/envs/transformers/lib/python3.8/site-packages/transformers/models/layoutlmv2/modeling_layoutlmv2.py", line 758, in _calc_text_embeddings
    embeddings = inputs_embeds + position_embeddings + spatial_position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (14) must match the size of tensor b (63) at non-singleton dimension 1

I'm investigating to fix the tensors problem.

jhubar · 2022-04-04T13:19:31Z

@chainyo have you find a solution for tensors problem ? 🤗

chainyo · 2022-04-04T17:57:09Z

@chainyo have you find a solution for tensors problem ? hugs

It seems that one tensor size is changing for no reason on my previous tests. I will dig more this week, I had no time last week.

jhubar · 2022-04-04T18:36:20Z

Great,if you want, we can organise a Google meet and looking the issue together. You can send me an email at : hubarjulien@gmail.com

sujit420 · 2022-04-15T11:09:16Z

@chainyo I am getting errors while using your PR during inference time. I am using the below code for token-classification (FUNSD):
processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")
image_path = '../funsd/page1.png'
image = Image.open(image_path).convert("RGB")
words, bboxes = get_words_and_boxes_textract(textract_client, image_path)
encoded_inputs = processor(image, words, boxes=bboxes, padding="max_length", truncation=True, return_tensors="pt")
for k,v in encoded_inputs.items():
encoded_inputs[k] = v.to(device)
dt = datasets.Dataset.from_dict(encoded_inputs)
outputs = loaded_ort_model.evaluation_loop(dt)

Error:
InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids

Any help is really appreciated. @chainyo @lewtun @michaelbenayoun

jhubar · 2022-04-15T11:32:47Z

You can try to add to the processor function token_type_ids like this :

  input_ids = batch['input_ids'].to(device)
  bbox = batch['bbox'].to(device)
  image = batch['image'].to(device)
  attention_mask = batch['attention_mask'].to(device)
  token_type_ids = batch['token_type_ids'].to(device)
  labels = batch['labels'].to(device)
  # forward pass
   outputs = model(input_ids=input_ids, bbox=bbox, image=image, attention_mask=attention_mask, 
                            token_type_ids=token_type_ids, labels=labels)

sujit420 · 2022-04-18T09:47:24Z

outputs = model(input_ids=input_ids, bbox=bbox, image=image, attention_mask=attention_mask,
token_type_ids=token_type_ids, labels=labels)

there is a method called evaluation_loop which does inferencing for loaded onnx models. it expects only huggingface dataset as per https://github.com/huggingface/optimum.

Please look at the snippet which I posted earlier.

github-actions · 2022-05-13T15:06:49Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

malcolmgreaves · 2022-10-04T21:25:40Z

Hi! Thanks for working on this @chainyo 🙏 I am interested in this work. Are you still working on this? If you no longer have the time or resources to do so, would you be able to provide any next steps as you see it for this ONNX export to work? Thank you for your time and effort 🤗

lewtun · 2022-10-05T08:16:46Z

Hey @chainyo, regarding your error with the tests - my guess is that the dummy data generation is the culprit. My suggestion would be to:

First pick an input that works with the torch model
Export the model to ONNX and check the forward pass still works with the same input
Generalise to the dummy input case

chainyo · 2022-10-05T08:16:57Z

@malcolmgreaves I don't even remember where I was with this issue months ago. But I will try to work on it this week if I can.

chainyo · 2022-10-05T08:18:19Z

@lewtun You are right! It seems that someone solved the things I was trying to achieve by doing it with LayoutLMv3. So I will check that and see what I can apply to v2.

github-actions · 2022-10-29T15:03:35Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

vibhas-singh · 2022-12-23T18:50:40Z

Hey @chainyo
Were you able to check this?
Any direction on this would be really helpful.

@lewtun You are right! It seems that someone solved the things I was trying to achieve by doing it with LayoutLMv3. So I will check that and see what I can apply to v2.

github-actions · 2023-01-17T15:03:27Z

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

Add LayoutLMv2 OnnxConfig

433c13e

chainyo mentioned this pull request Mar 21, 2022

ONNXConfig: Add a configuration for all available models #16308

Closed

chainyo and others added 2 commits March 23, 2022 16:30

add dummy bbox generation and return_bbox arguments

22aecfc

Merge branch 'main' into add-layoutlmv2-onnx-config

e368c1a

chainyo commented Mar 23, 2022

View reviewed changes

echarlaix mentioned this pull request Apr 15, 2022

need support for LayoutLMV2 models huggingface/optimum#143

Closed

github-actions bot closed this May 23, 2022

lewtun reopened this Oct 5, 2022

github-actions bot closed this Nov 6, 2022

lewtun reopened this Dec 24, 2022

github-actions bot closed this Jan 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add LayoutLMv2 OnnxConfig #16309

Add LayoutLMv2 OnnxConfig #16309

chainyo commented Mar 21, 2022 •

edited

HuggingFaceDocBuilderDev commented Mar 21, 2022

NielsRogge commented Mar 22, 2022

chainyo commented Mar 22, 2022

chainyo Mar 23, 2022

chainyo Mar 23, 2022

chainyo Mar 23, 2022

chainyo commented Mar 23, 2022 •

edited

jhubar commented Apr 4, 2022

chainyo commented Apr 4, 2022

jhubar commented Apr 4, 2022

sujit420 commented Apr 15, 2022 •

edited

jhubar commented Apr 15, 2022

sujit420 commented Apr 18, 2022

github-actions bot commented May 13, 2022

malcolmgreaves commented Oct 4, 2022 •

edited

lewtun commented Oct 5, 2022

chainyo commented Oct 5, 2022

chainyo commented Oct 5, 2022

github-actions bot commented Oct 29, 2022

vibhas-singh commented Dec 23, 2022

github-actions bot commented Jan 17, 2023

Add LayoutLMv2 OnnxConfig #16309

Add LayoutLMv2 OnnxConfig #16309

Conversation

chainyo commented Mar 21, 2022 • edited

What does this PR do?

Who can review?

HuggingFaceDocBuilderDev commented Mar 21, 2022

NielsRogge commented Mar 22, 2022

chainyo commented Mar 22, 2022

chainyo Mar 23, 2022

Choose a reason for hiding this comment

chainyo Mar 23, 2022

Choose a reason for hiding this comment

chainyo Mar 23, 2022

Choose a reason for hiding this comment

chainyo commented Mar 23, 2022 • edited

jhubar commented Apr 4, 2022

chainyo commented Apr 4, 2022

jhubar commented Apr 4, 2022

sujit420 commented Apr 15, 2022 • edited

jhubar commented Apr 15, 2022

sujit420 commented Apr 18, 2022

github-actions bot commented May 13, 2022

malcolmgreaves commented Oct 4, 2022 • edited

lewtun commented Oct 5, 2022

chainyo commented Oct 5, 2022

chainyo commented Oct 5, 2022

github-actions bot commented Oct 29, 2022

vibhas-singh commented Dec 23, 2022

github-actions bot commented Jan 17, 2023

chainyo commented Mar 21, 2022 •

edited

chainyo commented Mar 23, 2022 •

edited

sujit420 commented Apr 15, 2022 •

edited

malcolmgreaves commented Oct 4, 2022 •

edited