Skip to content

Commit

Permalink
Increase MNIST test tolerance (microsoft#20000)
Browse files Browse the repository at this point in the history
### Description

Found multiple occurrence of failures:

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1321061&view=logs&j=6df8fe70-7b8f-505a-8ef0-8bf93da2bac7&t=56a04c0b-9e7f-5c69-cb7b-c2a7b1a7392a&l=17537

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1329701&view=logs&j=6df8fe70-7b8f-505a-8ef0-8bf93da2bac7&t=4f6ef737-111d-50d1-a46b-5f86d9a970bc&s=3618b4c0-1011-591a-85b8-671e72e2cff1

1: [ RUN      ] ModelTests/ModelTest.Run/
cuda__models_zoo_opset7_MNIST_model
1: D:\a\_work\1\s\onnxruntime\test\providers\cpu\model_tests.cc(358):
error: Expected equality of these values:
1:   COMPARE_RESULT::SUCCESS
1:     Which is: 4-byte object <00-00 00-00>
1:   ret.first
1:     Which is: 4-byte object <01-00 00-00>
1: expected -2.33638 (c0158735), got -2.30239 (c0135a47), diff:
0.0339923, tol=0.0243638 idx=9
  • Loading branch information
tianleiwu committed Mar 21, 2024
1 parent 0b958bb commit 06fe4f3
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions onnxruntime/test/providers/cpu/model_tests.cc
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ TEST_P(ModelTest, Run) {
// when cuda or openvino is enabled, set it to a larger value for resolving random MNIST test failure
if (model_path.find(ORT_TSTR("_MNIST")) > 0) {
if (provider_name == "cuda" || provider_name == "openvino") {
per_sample_tolerance = 2.5e-2;
relative_per_sample_tolerance = 1e-2;
}
}
Expand Down

0 comments on commit 06fe4f3

Please sign in to comment.