Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for image classification #226

Merged
merged 7 commits into from
Jun 23, 2022

Conversation

regisss
Copy link
Contributor

@regisss regisss commented Jun 20, 2022

What does this PR do?

Fixes #173. This PR enables:

  • to use an image-classification pipeline
  • to perform quantization/optimization with computer vision models

At the moment, static post-training quantization with percentile/entropy calibration may not work with some models due to an issue with ONNX Runtime.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Copy link
Member

@philschmid philschmid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!🔥✅ I left a few comments. Mostly regarding breaking backward compatibility.
Other than this awesome PR! 🔥

optimum/onnxruntime/quantization.py Outdated Show resolved Hide resolved
docs/source/onnxruntime/modeling_ort.mdx Show resolved Hide resolved
docs/source/pipelines.mdx Show resolved Hide resolved
tests/onnxruntime/test_modeling_ort.py Show resolved Hide resolved
tests/onnxruntime/test_modeling_ort.py Show resolved Hide resolved
Copy link
Member

@michaelbenayoun michaelbenayoun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks a lot for adding this @regisss !
Left a few comments, but those are minor stuff, LGTM!

optimum/onnxruntime/modeling_ort.py Outdated Show resolved Hide resolved
optimum/onnxruntime/modeling_ort.py Outdated Show resolved Hide resolved
optimum/onnxruntime/optimization.py Show resolved Hide resolved
Copy link
Collaborator

@JingyaHuang JingyaHuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM(G means great here). :>

Just added some small nits.

docs/source/onnxruntime/modeling_ort.mdx Show resolved Hide resolved
docs/source/pipelines.mdx Outdated Show resolved Hide resolved
docs/source/pipelines.mdx Outdated Show resolved Hide resolved
optimum/onnxruntime/optimization.py Show resolved Hide resolved
@regisss regisss merged commit d8424cb into huggingface:main Jun 23, 2022
@regisss regisss deleted the add_computer_vision_support branch June 23, 2022 14:38
@davanstrien
Copy link
Member

Thanks so much for doing this -- really happy to see this feature added 🤗

@regisss
Copy link
Contributor Author

regisss commented Jun 23, 2022

Thanks so much for doing this -- really happy to see this feature added hugs

Do not hesitate to show us how you use it :) @davanstrien

@rhkenne
Copy link

rhkenne commented Jul 27, 2022

Thank you very much for the great work. The description of this PR mentioned "perform quantization/optimization with computer vision models". Does this include CLIP model (CLIPProcessor, CLIPModel) ? I am looking for an approach to serialize CLIP models (to onnx) and perform quantization/optimization. Any idea how ? @NielsRogge @JingyaHuang

@regisss
Copy link
Contributor Author

regisss commented Aug 2, 2022

Thank you very much for the great work. The description of this PR mentioned "perform quantization/optimization with computer vision models". Does this include CLIP model (CLIPProcessor, CLIPModel) ? I am looking for an approach to serialize CLIP models (to onnx) and perform quantization/optimization. Any idea how ? @NielsRogge @JingyaHuang

@rhkenne The first step would be to add ONNX support for CLIP in Transformers. Would you be interested in opening a PR to enable this?

Then, for optimization and quantization, additional work would be necessary because multimodal models are not supported yet.

@regisss
Copy link
Contributor Author

regisss commented Aug 8, 2022

@rhkenne There is an ongoing PR here to make CLIP exportable to ONNX.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for computer vision tasks
7 participants