Timm Models integration to Optimum-intel #404

sawradip · 2023-08-09T12:00:40Z

This PR is a part of Google Summer of Code 2023 project Showcase performance of PyTorch Image Models (Timm) with OpenVINO under Openvino-toolkit.

Goal

Main goal of this PR is seamless integration of Timm library(hosting 1100+ pytorch computer vision models, as of writing this) into the OpenVino ecosystem. And as Timm is now part of huggingface ecosystem, the integrations done through Optimum-intel.

Project Details and Mentors

Contributor: Sawradip Saha
Organization: OpenVino-toolkit
Size of project: 175 hours
Mentors: Alexander Kozlov, Liubov Talamanova

Background

Timm is a collection of pre-trained and optimized models for deep learning in computer vision. By providing a wide array(1100+ as of now) of state-of-the-art models with ease of use, it encourages research and development in the field of computer vision, making cutting-edge technology accessible to both professionals and enthusiasts.

OpenVINO is a toolkit designed to fast-track the development of high-performance computer vision and deep learning inference. By offering optimization across multiple hardware types, including CPUs, GPUs, and FPGAs, OpenVINO allows for efficient deployment in various environments.

The integration of Timm with OpenVINO combines the robust and accessible models from Timm with the high-performance and flexibility of OpenVINO. This synergy enables enhanced performance and scalability, making it an ideal solution for various applications ranging from research to production deployment.

Implementation Details

As most of Timm models are for Image-classification or feature-extraction, the integration is done through OVModelForImageClassification, so the user can load models like any other Huggingface models from HuggingFace Hub. Note that, previously the attempt to load the models in this ways raised a number of errors and unexpected behavious, similar to loading through AutoModelForImageClassification some of which are mentioned here.

Aditionally,

A feature extractor TimmImageProcessor has been implemented, as transformers didn't have any dedicated feature-extractor/image-precessor to handle the processor according to provided model config.
Several tests have been added, to test these new features.
Relevant changes to the docs.

Example:

With this PR we can load and use Timm models from hub like this:

import torch
from optimum.intel import OVModelForImageClassification
model_id = "timm/vit_tiny_patch16_224.augreg_in21k"
hf_model = OVModelForImageClassification.from_pretrained(model_id, export=True)
out = hf_model(pixel_values = torch.zeros((5, 3, hf_model.config.image_size, hf_model.config.image_size)))
print(out.logits.shape)

or

import requests
from PIL import Image
from optimum.intel.openvino.modeling_timm import TimmImageProcessor
from optimum.intel import OVModelForImageClassification

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
model_id = "timm/vit_tiny_patch16_224.augreg_in21k"
preprocessor = TimmImageProcessor.from_pretrained(model_id)
inputs = preprocessor(images=image, return_tensors='pt')
ov_model = OVModelForImageClassification.from_pretrained(model_id, export=True)
outputs = ov_model(pixel_values=inputs["pixel_values"])

AlexKoff88 · 2023-08-10T15:18:36Z

optimum/intel/openvino/modeling_timm.py

+    main_input_name = "pixel_values"
+
+
+class TimmModel(TimmPreTrainedModel):


I don't think we need such a model hierarchy, with TimmPreTrainedModel, TimmModel, etc. At least Optimum-Intel is not the place for such classes. What is required is just to put the logic of model creation and prefetch into the .from_timm() method of OVModelForTimm. Moreover, OVModelForTimm itself is not needed since there is no intent to expose it. It can be just a utility function or class that is not inherited from HF hierarchy at all.

I have removed the hierarchy as much as possible, leaving a small placeholder model to contain the timm models.

AlexKoff88 · 2023-08-10T15:19:11Z

optimum/intel/openvino/modeling_timm.py

+            return_dict=return_dict,
+        )
+
+        loss = None


Loss is not needed at all in the context of further export to OpenVINO.

I have removed that section.

AlexKoff88 · 2023-08-10T15:21:34Z

optimum/intel/openvino/modeling_timm.py

+        )
+
+
+class TimmImageProcessor(BaseImageProcessor, ImageFeatureExtractionMixin):


I didn't expect but this is great that you created it. I just wonder how much from the overall Timm preprocessing options this class supports and how we will validate it?

I also wonder. Though I have added and used the class, it would be better if we had any methodical way for validating.

Reduces abstractions

sawradip · 2023-08-12T15:47:41Z

@AlexKoff88 I have removed a of unnecessary code, and abstractions. Please take a look and let me know your feedback.

AlexKoff88 · 2023-08-13T07:26:25Z

optimum/intel/openvino/modeling_timm.py

+        return cls.from_dict(config_dict, **kwargs)
+
+
+class TimmOnnxConfig(ViTOnnxConfig):


You used ViTOnnxConfig as a base class here. Does it mean that only ViT models from Timm will work here?

I would inherit from VisionOnnxConfig to make it safer.

I have fixed that.

AlexKoff88 · 2023-08-13T07:30:15Z

tests/openvino/test_modeling.py

@@ -619,6 +623,27 @@ def test_pipeline(self, model_arch):
        self.assertTrue(isinstance(outputs[0]["label"], str))
        gc.collect()

+    @parameterized.expand(TIMM_MODELS)


I think we should have a test which:

load a timm model from hub

saves it in IR locally (.save_pretrained)

loads IR and does inference (random data is ok)

A test test_timm_save_and_infer is added.

AlexKoff88 · 2023-08-16T08:22:39Z

@fxmarty, @echarlaix, can you please take a look?

HuggingFaceDocBuilderDev · 2023-08-17T05:55:07Z

The documentation is not available anymore as the PR was closed or merged.

fxmarty

I left a few comments for tiny things to be fixed, but otherwise looks good to me, thank you!

It could be worth mentioning in the documentation as well that OVModelForImageClassification can handle timm models.

optimum/intel/openvino/modeling_diffusion.py

fxmarty · 2023-08-17T07:21:59Z

optimum/intel/openvino/modeling_timm.py

+class TimmOnnxConfig(VisionOnnxConfig):
+    DEFAULT_TIMM_ONNX_OPSET = 13
+    outputs = OrderedDict([("logits", {0: "batch_size"})])
+    NORMALIZED_CONFIG_CLASS = NormalizedVisionConfig
+    MIN_TORCH_VERSION = version.parse("1.11")
+
+    @property
+    def inputs(self) -> Dict[str, Dict[int, str]]:
+        return {"pixel_values": {0: "batch_size", 1: "num_channels", 2: "height", 3: "width"}}


Is this to support the ONNX export of timm models? Do you think it would be useful to move it (later) in optimum main repo?

Thanks, @fxmarty for the review. It could be useful in case ORT supports such models as well. We are moving to direct PyTorch import of model to OpenVINO as you can see in #397. So, this part will be definitely revised after PR #397 is merged. So far, I would keep it as is.

Sounds good!

optimum/intel/openvino/modeling_timm.py

fxmarty · 2023-08-17T11:05:59Z

It could be worth mentioning in the documentation as well that OVModelForImageClassification can handle timm models.

@sawradip Would you like to precise the doc / add an example, or should I merge as is?

sawradip · 2023-08-17T11:18:16Z

I will add some refernce to the doc as well add some description to the PR.

AlexKoff88 · 2023-08-18T10:01:19Z

@fxmarty, I see the following error in two PRs in a raw: "cannot import name 'is_safetensors_available' from 'diffusers.utils''. Do you have any idea how to fix it quickly? Do we need to update the requirements?

…intel into integrate_timm

sawradip · 2023-08-21T20:42:56Z

@fxmarty , I have done some additions in docs, as well as added description to this PR. If everything seems alright, you can merge.

…intel into integrate_timm

sawradip · 2023-08-22T09:29:02Z

@fxmarty There was one error with style, fixed it.

Any feedback?

fxmarty

LGTM!

echarlaix · 2023-08-23T17:48:06Z

setup.py

@@ -17,6 +17,7 @@
    "datasets>=1.4.0",
    "sentencepiece",
    "scipy",
+    "timm",


I don't think timm should become a hard dependency

sawradip added 4 commits August 9, 2023 17:42

Relevant file changes

ceda790

Added test for timm

34d8c3d

Fixed Style

bf3b7db

Added style changes

8beba03

AlexKoff88 reviewed Aug 10, 2023

View reviewed changes

sawradip and others added 4 commits August 12, 2023 21:29

Reduces abstractions

bf6e017

Merge pull request #1 from sawradip/integrate_timm_testing

656b05b

Reduces abstractions

Fixed styles

2adcb1b

Merge branch 'integrate_timm_testing' into integrate_timm

0556247

AlexKoff88 reviewed Aug 13, 2023

View reviewed changes

Added Feature & text for Timm model saving

1303c97

AlexKoff88 changed the title ~~Timm Models integration to Optimum-intel[Draft]~~ [Draft]:Timm Models integration to Optimum-intel Aug 14, 2023

AlexKoff88 changed the title ~~[Draft]:Timm Models integration to Optimum-intel~~ [Draft]: Timm Models integration to Optimum-intel Aug 14, 2023

AlexKoff88 changed the title ~~[Draft]: Timm Models integration to Optimum-intel~~ Timm Models integration to Optimum-intel Aug 14, 2023

Added Timm to dependency

ecedcde

fxmarty approved these changes Aug 17, 2023

View reviewed changes

sawradip and others added 4 commits August 17, 2023 14:28

Added Image Processor source reference

2458b03

Merge branch 'huggingface:main' into integrate_timm

52b03e2

Fixed Divergence in modeling_diffusion.py

b0f1d66

Fixed divergence in modeling_diffusion.py

988fa4e

eaidova mentioned this pull request Aug 18, 2023

Fix import error is_safetensors_available #407

Merged

sawradip added 2 commits August 22, 2023 01:28

Adde Timm example to docs

d3cff5f

Merge branch 'integrate_timm' of https://github.com/sawradip/optimum-…

9dadb9a

…intel into integrate_timm

sawradip and others added 3 commits August 22, 2023 02:47

Merge branch 'huggingface:main' into integrate_timm

b463c41

Fixed style

a2b3d38

Merge branch 'integrate_timm' of https://github.com/sawradip/optimum-…

6f3c28c

…intel into integrate_timm

fxmarty approved these changes Aug 23, 2023

View reviewed changes

fxmarty merged commit 0760d9a into huggingface:main Aug 23, 2023
6 of 10 checks passed

echarlaix reviewed Aug 23, 2023

View reviewed changes

echarlaix mentioned this pull request Aug 24, 2023

Fix timm ov model #413

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Timm Models integration to Optimum-intel #404

Timm Models integration to Optimum-intel #404

sawradip commented Aug 9, 2023 •

edited

AlexKoff88 Aug 10, 2023

sawradip Aug 12, 2023

AlexKoff88 Aug 10, 2023

sawradip Aug 12, 2023

AlexKoff88 Aug 10, 2023

sawradip Aug 14, 2023

sawradip commented Aug 12, 2023

AlexKoff88 Aug 13, 2023

AlexKoff88 Aug 13, 2023

sawradip Aug 14, 2023

AlexKoff88 Aug 13, 2023

sawradip Aug 14, 2023

AlexKoff88 commented Aug 16, 2023

HuggingFaceDocBuilderDev commented Aug 17, 2023 •

edited

fxmarty left a comment

fxmarty Aug 17, 2023

AlexKoff88 Aug 17, 2023

fxmarty Aug 17, 2023

fxmarty commented Aug 17, 2023

sawradip commented Aug 17, 2023

AlexKoff88 commented Aug 18, 2023

sawradip commented Aug 21, 2023

sawradip commented Aug 22, 2023

fxmarty left a comment

echarlaix Aug 23, 2023

		main_input_name = "pixel_values"


		class TimmModel(TimmPreTrainedModel):

		)


		class TimmImageProcessor(BaseImageProcessor, ImageFeatureExtractionMixin):

		return cls.from_dict(config_dict, **kwargs)


		class TimmOnnxConfig(ViTOnnxConfig):

Timm Models integration to Optimum-intel #404

Timm Models integration to Optimum-intel #404

Conversation

sawradip commented Aug 9, 2023 • edited

Goal

Project Details and Mentors

Background

Implementation Details

Example:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawradip commented Aug 12, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexKoff88 commented Aug 16, 2023

HuggingFaceDocBuilderDev commented Aug 17, 2023 • edited

fxmarty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fxmarty commented Aug 17, 2023

sawradip commented Aug 17, 2023

AlexKoff88 commented Aug 18, 2023

sawradip commented Aug 21, 2023

sawradip commented Aug 22, 2023

fxmarty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sawradip commented Aug 9, 2023 •

edited

HuggingFaceDocBuilderDev commented Aug 17, 2023 •

edited