Gather node discrepancy for out of index indices

[romain-keramitas-prl]

Describe the bug
According to the documentation, indices sent to a Gather node must be in the range [-s, s -1] where s is the first dimension of the data input. An error is indeed raised when the CPUExecutionProvider is used, however that is not the case when the CUDAExecutionProvider is used.

Urgency
Low, we ran into this problem when converting some unit tests that were used with the original model. This may result in silent errors, but in most cases should not be an issue.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• ONNX Runtime installed from (source or binary): binary (wheel)
• ONNX Runtime version: 1.10
• Python version: 3.9.9
• Visual Studio version (if applicable): NA
• GCC/Compiler version (if compiling from source): NA
• CUDA/cuDNN version: 11.4
• GPU model and memory: GeForce RTX 2060, 6GB

To Reproduce

See gist here.

Expected behavior
An error to be raised as well when using the CUDAExecutioonProvider

Screenshots
NA

Additional context
NA

[tianleiwu]

Looks like an issue of CUDA implementation of Gather op: miss input validation.

[hariharans29]

This is a behavior adopted by some other frameworks as well: https://www.tensorflow.org/api_docs/python/tf/gather

"On CPU, if an out of bound index is found, an error is raised. On GPU, if an out of bound index is found, a 0 is stored in the corresponding output value."

The ONNX spec does say : "It is an error if any of the index values are out of bounds." Given that, we may have to do some validations on CUDA as well.

[stale]

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[snnn]

This issue has been automatically closed as 'not planned' because it has been marked as 'stale' for more than 30 days without activity. If you believe this is still an issue, please feel free to reopen it.