Temporarily fix ONNX model exporting error

[SatyaJandhyalaAtMS]

…issues/143

What does this PR do?

Fix the following error while trying to export ONNX model
TypeError: '<=' not supported between instances of 'tuple' and 'Tensor'

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

cc @amyeroberts

[amyeroberts]

Thanks for opening this PR @SatyaJandhyalaAtMS !

Could you share a link to the issue this resolves? I'm getting a 404 error for the link in the commit / PR title: https://github.com/microsoft/onnx-converters-private/issues/143

[SatyaJandhyalaAtMS]

The python code to reproduce the error is:

import onnxruntime as ort
from transformers import AutoImageProcessor, AutoModelForImageClassification
from PIL import Image
import requests
import numpy as np
import torch.onnx

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
processor = AutoImageProcessor.from_pretrained("microsoft/swinv2-tiny-patch4-window8-256")
model = AutoModelForImageClassification.from_pretrained("microsoft/swinv2-tiny-patch4-window8-256")
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits
# model predicts one of the 1000 ImageNet classes
predicted_class_idx = logits.argmax(-1).item()
options = ort.SessionOptions()
# options.log_severity_level = 0
torch.onnx.export(model, inputs['pixel_values'],"swinv2.onnx", export_params=True, opset_version=11, do_constant_folding=True, input_names=["input"], output_names=["output"])
ort_sess = ort.InferenceSession("swinv2.onnx", providers=["CUDAExecutionProvider"], sess_options=options)
ort_outputs=ort_sess.run(None, {"input":inputs['pixel_values'].numpy()})
ort_prediction=int(np.argmax(np.array(ort_outputs[0]).squeeze(), axis=0))
if ort_prediction == predicted_class_idx:
    print("Test passed")
else:
    print("Test failed")

The error is as follows:

Traceback (most recent call last):
File "test_with_ort.py", line 19, in
torch.onnx.export(model, inputs['pixel_values'],"swinv2.onnx", export_params=True, opset_version=11, do_constant_folding=True, input_names=["input"], output_names=["output"])
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/onnx/utils.py", line 504, in export
_export(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/onnx/utils.py", line 1529, in _export
graph, params_dict, torch_out = _model_to_graph(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/onnx/utils.py", line 1111, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/onnx/utils.py", line 987, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/onnx/utils.py", line 891, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/jit/_trace.py", line 1184, in _get_trace_graph
outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward
graph, out = torch._C._create_graph_by_tracing(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper
outs.append(self.inner(*trace_inputs))
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.py", line 1274, in forward
outputs = self.swinv2(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.py", line 1078, in forward
encoder_outputs = self.encoder(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.py", line 907, in forward
layer_outputs = layer_module(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.py", line 821, in forward
layer_outputs = layer_module(
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1182, in _slow_forward
result = self.forward(*input, **kwargs)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.py", line 723, in forward
self.set_shift_and_window_size(input_dimensions)
File "/home/sajandhy/miniconda3/envs/py38/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.py", line 670, in set_shift_and_window_size
if input_resolution
TypeError: '<=' not supported between instances of 'tuple' and 'Tensor'

[SatyaJandhyalaAtMS]

My Python environment is
onnxruntime 1.14.1
torch 1.13.1+cu116
torchaudio 0.13.1+cu116
torchvision 0.14.1+cu116
transformers 4.26.1
My platform is
Ubuntu 22.04

[amyeroberts]

Thanks for adding this fix @SatyaJandhyalaAtMS and for all the repro info!

Looking into the exporting error, the issue is coming from this line, where window_size is being reset from input_resolution which is a tensor. Could you instead make a similar .item() check there upstream to make sure self.window_size has the expected type throughout

i.e.

        window_dim = input_resolution[0].item() if torch.is_tensor(input_resolution[0]) else input_resolution[0]
        self.window_size = (
            window_dim if window_dim <= target_window_size[0] else target_window_size[0]
        )

[amyeroberts]

Thanks for updating @SatyaJandhyalaAtMS !

The tests are currently failing on the code quality checks. As the lines of code that have been modified are in a class with a # Copied from header, the original code source the comment points to will need to be updated i.e. the equivalent line in transformers.models.swin.modeling_swin.SwinOutput.

Then run make fix-copies to propogate the change across the repo.

To get the other styling tests pass, run make style to have the code formatted in the expected style

[amyeroberts]

Thanks for adding this fix!

[sgugger]

Thanks for the fix!