BirdClassifierTest.test_compile_tflite failing in regression tests

[julianwa]

https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true

FAIL: TFLITEHUB :: bird_classifier_test.py (3 of 53)
******************** TEST 'TFLITEHUB :: bird_classifier_test.py' FAILED ********************
Script:
--
: 'RUN: at line 1';   /opt/hostedtoolcache/Python/3.9.13/x64/bin/python /home/runner/work/iree-samples/iree-samples/tflitehub/bird_classifier_test.py
--
Exit Code: 1
Command Output (stdout):
--
$ ":" "RUN: at line 1"
$ "/opt/hostedtoolcache/Python/3.9.13/x64/bin/python" "/home/runner/work/iree-samples/iree-samples/tflitehub/bird_classifier_test.py"
# command output:
TMP_DIR = /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py
# command stderr:
2022-06-03 15:07:26.094638: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.[11](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:12).0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.[13](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:14)/x64/lib
2022-06-03 15:07:26.094679: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Running tests under Python 3.9.13: /opt/hostedtoolcache/Python/3.9.13/x64/bin/python
2022-06-03 15:07:28.588283: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
2022-06-03 15:07:28.588377: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-06-03 15:07:28.588415: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (fv-az58-885): /proc/driver/nvidia/version does not exist
[ RUN      ] BirdClassifierTest.test_compile_tflite
I0603 15:07:28.784186 [14](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:15)0321084335936 test_util.py:119] Setting up for IREE
I0603 [15](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:16):07:28.785079 140321084335936 binaries.py:2[18](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:19)] Invoke IREE Pipeline:
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/tools/tflite/iree-import-tflite /home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/model.tflite --mlir-print-debuginfo --save-temp-tfl-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/tflite.mlir --save-temp-iree-input=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/tosa.mlir
  /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-compile - --iree-input-type=tosa --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=dylib-llvm-aot -o=/home/runner/work/iree-samples/iree-samples/tflitehub/tmp/bird_classifier_test.py/module.bytecode --iree-mlir-to-vm-bytecode-module --iree-llvm-embedded-linker-path=/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/iree/compiler/tools/../_mlir_libs/iree-lld --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false
I0603 15:07:58.786024 1403[21](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:22)084335936 test_util.py:75] Setting up tflite interpreter
I0603 15:07:58.787774 140321084335936 test_util.py:82] Setting up iree runtime
I0603 15:07:58.795829 140321084335936 test_util.py:131] Setting up test inputs
I0603 15:07:59.056051 140321084335936 test_util.py:134] Invoking TFLite
I0603 15:07:59.084538 140321084335936 test_util.py:96] Invocation time: 0.0280 seconds
I0603 15:07:59.084826 140321084335936 test_util.py:137] Invoke IREE
I0603 15:07:59.[24](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:25)4467 140[32](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:33)1084[33](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:34)5936 test_util.py:111] Invocation time: 0.1594 seconds
I0603 15:07:59.245091 1403210843[35](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:36)9[36](https://github.com/google/iree-samples/runs/6727972570?check_suite_focus=true#step:6:37) test_util.py:72] Max error (0): 4.000000
[  FAILED  ] BirdClassifierTest.test_compile_tflite

[jpienaar]

So this is regression in numerical accuracy?

[jpienaar]

These seem to both be failing since 38b0bd3 (not necessarily related, just when test was run where it failed)

[jpienaar]

Oh sorry, misread. Failing at

iree-compiler==20220516.140
iree-runtime==20220516.140
iree-tools-tf==20220516.140
iree-tools-tflite==20220516.140
iree-tools-xla==20220516.140
tensorflow==2.9.0
tensorflow-estimator==2.9.0
tensorflow-io-gcs-filesystem==0.25.0

Passing before

iree-compiler==20220516.140
iree-runtime==20220516.140
iree-tools-tf==20220516.140
iree-tools-tflite==20220516.140
iree-tools-xla==20220516.140
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.25.0