GridSample ND

[leimao]

Description

Extending the existing GridSample (2D) operator to any arbitrary dimensions.

This PR is ready for review.

Motivation and Context

PyTorch GridSample supports both 2D and 3D interpolation whereas ONNX does not.

#4779

TODOs

1. The PyTorch GridSample PR has been merged. Update the ONNX unit test reference values using the PyTorch TOT reference outputs.
2. Move the GridSample from Opset 19 to Opset 20.
3. Create Adapters for GridSample version upgrades.

[leimao]

@xadupre I realized that the TOT ONNX CI has added ONNX Runtime unit tests. I think this is inappropriate. Why ONNX which is something that ONNX Runtime depends on needs to be tested against ONNX Runtime?
In addition, because I extended the GridSample operator spec, ONNX Runtime does not support this operator anymore. How could I disable the ONNX Runtime tests for GridSample? Thank you.

[xadupre]

You can add a section here https://github.com/onnx/onnx/blob/main/onnx/test/test_backend_onnxruntime.py#L231 for opset 20 and disable the tests you need.

[leimao]

@gramalingam @xadupre Some of the CI tests are taking forever. But if you check the details, they were actually successful. Do you know why?

[gramalingam]

@gramalingam @xadupre Some of the CI tests are taking forever. But if you check the details, they were actually successful. Do you know why?

Hmmm ... I don't know. @jcwchen , do you know? Thanks!

[jcwchen]

Hmmm ... I don't know. @jcwchen , do you know? Thanks!

I haven't bumped into such kind of situation before. I guess AzurePipelines was downgraded at that moment. Rerun should help. Please let me know if this problem persists. Thanks!

[leimao]

Hmmm ... I don't know. @jcwchen , do you know? Thanks!

I haven't bumped into such kind of situation before. I guess AzurePipelines was downgraded at that moment. Rerun should help. Please let me know if this problem persists. Thanks!

Thanks. I will push another commit after I address the reviewers' comments.

[yang-Michael]

@leimao hello, I really admire the countless contributions you've made to the pytorch and onnx code, including the gridsample operator. So, I'm dying to know when will the 5D input supported version may be released？

[leimao]

@leimao hello, I really admire the countless contributions you've made to the pytorch and onnx code, including the gridsample operator. So, I'm dying to know when will the 5D input supported version may be released？

This PR is being reviewed now. Hopefully it can be merged to main soon. But the official Opset 20 release, which includes the GridSample ND, will be in roughly 3 months from today I think. Once this PR is merged to main, the backend runtimes, such as ONNX Runtime and TensorRT, can start to work on supporting GridSample ND.

[liqunfu]

@leimao , thank you for contributing to ONNX. Now I am preparing ONNX 1.15.0 release and integrating the release with ORT, I wonder if you have a plan to implement the runtime kernel for this updated op in ORT? If so, I will let you know once ORT has ONNX 1.15.0 integrated. Thank you in advance.

[leimao]

@leimao , thank you for contributing to ONNX. Now I am preparing ONNX 1.15.0 release and integrating the release with ORT, I wonder if you have a plan to implement the runtime kernel for this updated op in ORT? If so, I will let you know once ORT has ONNX 1.15.0 integrated. Thank you in advance.

@liqunfu Thanks for bringing this up. For the moment, I probably don't have the bandwidth to commit a runtime kernel.