Pad behavior is inconsistent with ONNX ReferenceEvaluator and documentation

[vincentme]

behavior

In mode 'reflect', the result from onnxscript and onnxruntime is inconsistent with onnx.reference.ReferenceEvaluator, onnx documentation, and numpy.pad.

In the eager mode of onnxscript (res_onnxscript2), the computation is indeed conduct by onnxruntime, right? But is the res_onnxscript1 from onnxscript evaluator? I don't know much about onnxscript internal. But the result from onnxruntime is defenitely inconsistent, so I also created an issue there.

code to reproduce

import numpy as np
from onnxscript import script, FLOAT, opset20 as op
import onnx

data = np.array([
    [1.0, 1.2],
    [2.3, 3.4],
    [4.5, 5.7],
], dtype = np.float32)

mode = 'reflect'

res_np = np.pad(data, ((0, 0), (2, 0)), mode = mode)
res_onnxscript1 = op.Pad(data, pads = [0, 2, 0, 0], mode = mode)

@script()
def test(data: FLOAT[None, None]) -> FLOAT[None, None]:
    return op.Pad(data, pads = [0, 2, 0, 0], mode = mode)
res_onnxscript2 = test(data)

model = test.to_model_proto()
feeds = {'data': data,}

from onnx.reference import ReferenceEvaluator
sess = ReferenceEvaluator(model)
outpus_onnx = sess.run(None, feeds)

from onnxruntime import InferenceSession
sess = InferenceSession(model.SerializeToString())
outpus_onnxruntime = sess.run(None, feeds)

print(f'numpy:\n{res_np}\n')
print(f'onnxscript evaluator:\n{res_onnxscript1}\n')
print(f'onnxscript evaluator (called onnxruntime?):\n{res_onnxscript2}\n')
print(f'onnx ReferenceEvaluator:\n{outpus_onnx[0]}\n')
print(f'onnxruntime:\n{outpus_onnxruntime[0]}\n')

print(onnx.printer.to_text(model))

result

numpy:
[[1.  1.2 1.  1.2]
 [2.3 3.4 2.3 3.4]
 [4.5 5.7 4.5 5.7]]

onnxscript evaluator:
Tensor(array([[0. , 1.2, 1. , 1.2],
       [0. , 3.4, 2.3, 3.4],
       [0. , 5.7, 4.5, 5.7]], dtype=float32))

onnxscript evaluator (called onnxruntime?):
[[0.  1.2 1.  1.2]
 [0.  3.4 2.3 3.4]
 [0.  5.7 4.5 5.7]]

onnx ReferenceEvaluator:
[[1.  1.2 1.  1.2]
 [2.3 3.4 2.3 3.4]
 [4.5 5.7 4.5 5.7]]

onnxruntime:
[[0.  1.2 1.  1.2]
 [0.  3.4 2.3 3.4]
 [0.  5.7 4.5 5.7]]

<
   ir_version: 9,
   opset_import: ["" : 20]
>
test (float[?,?] data) => (float[?,?] return_val) {
   const = Constant <value: tensor = int64[4] const {0,2,0,0}> ()
   return_val = Pad <mode: string = "reflect"> (data, const)
}

[justinchuby]

Looks like ONNX Runtime is not following the spec to me. @gramalingam does it look like it for you?

[gramalingam]

Yes, it looks like a bug in onnxruntime.