Add ONNX export for ViTMatte models #1582
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Still an issue with shapes other than what the model is exported with: Running it with input of shape [1, 1, 64, 92] gives: (but it works in pytorch) |
|
Hi, Sorry I'm not an ONNX expert so not sure how those lines should be updated. Might be better to ping someone from the Optimum team |
|
@xenova If you add a test with a tiny model we can merge this! |
|
I was looking on the Hub for a tiny random model, but I couldn't find one (so I skipped adding the test). If you'd like, I can add https://huggingface.co/hustvl/vitmatte-small-composition-1k (25M params). However, there's still an issue with python casts ( Will convert to draft while we discuss this. |
|
@xenova Thank you so much for the contribution. Since I'm quite new to ONNX. Can you please give me an example to inference your exported ONNX model here in Python? Here was what I did: However, this gave me: In addition, I wonder if this can be done: Thank you so much for your help! |
|
After digging deeper into this, I possibly found a proper way to inference the exported model. Yet still got error.
However, executing the above code returned: The inference in JS by @xenova available here seems very straight-to-the-point. However, since I'm not much familiar with JS, I can't understand what he did under the hood, yet I belive there should be a similar and simple way to do it in Python. Can someone please help me with this? Thank you so much! |
|
@EricLe-dev You were very close! Here's some example code: import onnxruntime
from PIL import Image
import numpy as np
from transformers import VitMatteImageProcessor
from huggingface_hub import hf_hub_download
ort_sess = onnxruntime.InferenceSession(
"./model.onnx", providers=['CPUExecutionProvider']
)
filepath = hf_hub_download(
repo_id="hf-internal-testing/image-matting-fixtures", filename="image.png", repo_type="dataset"
)
image = Image.open(filepath).convert("RGB")
filepath = hf_hub_download(
repo_id="hf-internal-testing/image-matting-fixtures", filename="trimap.png", repo_type="dataset"
)
trimap = Image.open(filepath).convert("L")
processor = VitMatteImageProcessor.from_pretrained("hustvl/vitmatte-small-composition-1k")
inputs = processor(images=image, trimaps=trimap, return_tensors="pt")
outputs = ort_sess.run(None, {'pixel_values' : inputs['pixel_values'].numpy()})
alphas = outputs[0]
# Visualize
result = Image.fromarray(np.uint8(alphas[0][0] * 255), mode='L')
resultProduces: |
|
@xenova Thank you so much for your answer. I encountered an another issue that I would very appreciate if you can help me. My current system is: The different part here was, my system has CUDA 11.3 and I could not install any Pytorch version that is newer than 1.11.0+cu113 With the mentioned system, when I run: It would complain that: In order to get it running, I did a little trick, which are: Doing all of these trick allowed me to run the command to export the model. Inferencing the exported model with your code snippet gave me this error: Does it has anything to do with the version of Transformers? or the version of PyTorch (since you are using Torch 2.1.1+cu121 and Python 3.10)? Can you please share your full pip list? Your help is much appreciated! |
|
@xenova After install onnxruntime-gpu==1.14.0, then ran the following command:
I got exactly the same output as yours, which is:
Running this piece of code with your exported model work perfectly. However, running it with my exported model gave this error: Using Netron, I was able to see the difference between your exported model and my exported model. I belive that in order to get the code running, you did change some layers in the model. Can you please confirm this and give me some more ideas how to get it work? Thank you so much! |
|
@xenova Here is my pip list: I'm running the code with And this is my nvidia-smi, my CUDA version is 11.4 but I could install the Torch 2.1.1+cu118 and it still works so I'm not sure if this is an issue. Exporting the model with this environment gave me a slightly different model (viewed in Netron), yet still missing the Pow and some other layers compared to yours. This indicating that there is something wrong in my env. Can you please share the full environment that you used to export the model? Thank you so much! |
|
As mentioned here, I did need to update some of the transformers modelling code to get the export working. In particular, you need to watch out for warnings that mention converting to a python float/integer, and then those calls to |
|
@xenova Thank you for your reply. Here was my output: I believe what you meant was editing the lines mentioned in those warnings to
Does this what you did? Thank you so much! |
|
After diving in thousand lines of code (using vscode, searching for direct Python casts of
My export output from is exactly as yours, which are: There are warnings in this output, most of them are about converting a tensor to a Python boolean. There are 2 lines of warning mentioned something relating to
@xenova can you please point me to the right file or line of code that you modified? Thank you a ton! |
|
Let's merge this once huggingface/transformers#30065 is released |
|
question is where the hell do i get the model? |
What does this PR do?
As title says :)
I haven't added a test yet, since I couldn't find a tiny random model on the HF hub.
Fixes # (issue)
Before submitting