Support for converting LayoutLM to ONNX

[nishprabhu]

🚀 Feature request

Transformers currently provides ready configurations for converting BERT, BART, RoBERTa and several other models to ONNX. Can we extend this to also support LayoutLM?

Motivation

ONNX is quickly becoming the default runtime environment in many production settings. Ideally, all models supported by the library should have an easy path to conversion.

Your contribution

I am willing to submit a PR that implements this.

[NielsRogge]

Sure that would be great. LayoutLM is literally only adding 4 additional embedding layers to BERT:

transformers/src/transformers/models/layoutlm/modeling_layoutlm.py

Lines 66 to 69 in a3f96f3

	self.x_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.hidden_size)
	self.y_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.hidden_size)
	self.h_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.hidden_size)
	self.w_position_embeddings = nn.Embedding(config.max_2d_position_embeddings, config.hidden_size)

So I guess it won't be that difficult to support?

cc @mfuntowicz

[NielsRogge]

The guide written here is very helpful: https://huggingface.co/transformers/serialization.html?highlight=onnx#converting-an-onnx-model-using-the-transformers-onnx-package

[nishprabhu]

Thanks! It was very useful!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[d-v-dlee]

dumb question - how do i format the ORT inputs for LayoutLM onnx? does anyone have an example of LayoutLM ONNX inference?

I'm trying to pass in the output of a collator into the onxx session. Its not liking the bounding box tensor since it its dimensions are different than input_ids, token_type_ids and attention_mask.