Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an example of exporting BartModel + BeamSearch to ONNX module. #13765

Merged
merged 6 commits into from
Oct 7, 2021
Merged

Add an example of exporting BartModel + BeamSearch to ONNX module. #13765

merged 6 commits into from
Oct 7, 2021

Conversation

fatcat-z
Copy link
Contributor

This PR is aimed to deliver an example to describe how to export BartModel + BeamSearch to ONNX model and inference it by ONNXRuntime.

These new example files were added without updates to existing Huggingface model code. It mainly focuses on:

  1. Providing an entry file run_onnx_exporter.py to run this example.
  2. In generation_onnx.py file, adding a BARTGenerator model to wrap the native BartModel code. To support Beam Search, we need to convert the whole model to a TorchScript module finally. This new BARTGenerator model will trace the Encoder and Decoder of BartModel to make them be compatible with torch.jit.script() method.
  3. In generation_onnx.py file, copying Beam Search code from generation_utils.py and updating these code so that beam_search function could be converted to TorchScript by calling torch.jit.script method.

@fatcat-z fatcat-z changed the title To onnx example Add an example of exporting BartModel + BeamSearch to ONNX module. Sep 28, 2021
@patrickvonplaten
Copy link
Contributor

Think we just need to run make style once and then we can merge - lgtm!

@mfuntowicz mfuntowicz merged commit 279ce5b into huggingface:master Oct 7, 2021
@NielsRogge
Copy link
Contributor

Thanks for adding this! Could you perhaps add a README to this example, to explain what the example is about and how to use it?

Copy link
Contributor

@garymm garymm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this @fatcat-z.
Could you please send another PR that adds a README and addresses my comments?

@NielsRogge
Copy link
Contributor

Hey @fatcat-z,

kindly pinging you for taking into account the comments on this PR (perhaps in a follow-up PR), if possible!

@fatcat-z
Copy link
Contributor Author

Thanks for adding this! Could you perhaps add a README to this example, to explain what the example is about and how to use it?

I'm working on a new PR to address your and Gary's comments.

Albertobegue pushed a commit to Albertobegue/transformers that referenced this pull request Jan 27, 2022
…uggingface#13765)

* Add all example files.

* Reformat files by black.

* Style.

* Remove unused imports.

Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants