Incorrect output for unit test example in tensorflow/tensorflow/lite/micro/examples/hello_world/hello_world_test.cc

[spartacoos]

@tensorflow/micro

System information
Laptop Model: Inspiron-7370
OS: Ubuntu 18.04.5 LTS x86_64
Host: Inspiron 7370
Kernel: 5.4.0-42-generic
Uptime: 4 hours, 48 mins
Packages: 2256
Shell: bash 4.4.20
Resolution: 1920x1080, 1920x1080
DE: GNOME 3.28.4
WM: GNOME Shell
WM Theme: Adwaita
Theme: Ambiance [GTK2/3]
Icons: Ubuntu-mono-dark [GTK2/3]
Terminal: gnome-terminal
CPU: Intel i7-8550U (8) @ 4.000GHz
RAM: 8GB
GPU: Intel UHD Graphics 620
Memory: 4790MiB / 7573MiB

• TensorFlow installed from (source or binary): git clone https://github.com/tensorflow/tensorflow.git
• Tensorflow version (commit SHA if source):
• Target platform (e.g. Arm Mbed OS, Arduino Nano 33 etc.): Mbed OS board (but this is irrelevant since this test runs locally)

Describe the problem
When I make a change in the hello_world_test.cc file such as for example changing a value to something that I know should return an error such as checking:

input->data.f[0] = 4.f;
  invoke_status = interpreter.Invoke();
  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, invoke_status);

  value = output->data.f[0];
  TF_LITE_MICRO_EXPECT_NEAR(-0.959f, value, 0.05f);

when I run this test

run make -f tensorflow/lite/micro/tools/make/Makefile test_hello_world_test 

I should obtain:

Testing LoadModelAndPerformInference
[whatever this value is] near value failed at tensorflow/lite/micro/examples/hello_world/hello_world_test.cc:94
0/1 tests passed
~~~SOME TESTS FAILED~~~

Instead, I obtain the following:

~~~ALL TESTS PASSED~~~'
tensorflow/lite/micro/examples/hello_world/Makefile.inc:33: recipe for target 'test_hello_world_test' failed
make: *** [test_hello_world_test] Error 1

In other words, the "tensorflow/lite/micro/testing/micro_test.h" framework incorrectly asserts if tests have passed and prints "ALL TESTS PASSED" when some tests have in fact failed. Also, it doesn't specify which tests failed (or the line in hello_world_test.cc). As seen in the error message this is probably a problem with the Makefile.

Please provide the exact sequence of commands/steps when you ran into the problem

me$ git clone https://github.com/tensorflow/tensorflow.git
me$ ls 
/tensorflow
me$ cd tensorflow/tensorflow/lite/micro/examples/hello_world/
me$ nano hello_world_test.cc 

Changed the test comparison values at the bottom to make them fail on purpose i.e

input->data.f[0] = 4.f; // this was "input->data.f[0] = 5.f;" so setting it to 4 should cause an error as sin(4) != sin(5)
  invoke_status = interpreter.Invoke();
  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, invoke_status);

  value = output->data.f[0];
  TF_LITE_MICRO_EXPECT_NEAR(-0.959f, value, 0.05f);

me$ cd ~
me$ cd tensorflow
me$ run make -f tensorflow/lite/micro/tools/make/Makefile test_hello_world_test 
~~~ALL TESTS PASSED~~~'
tensorflow/lite/micro/examples/hello_world/Makefile.inc:33: recipe for target 'test_hello_world_test' failed
make: *** [test_hello_world_test] Error 1

[advaitjain]

Confirming that this issue is reproducible.

The root cause is the -e flag to bash at this line.

Modifying hello_world_test.cc to cause an error and then running the following command:

tensorflow/lite/micro/testing/test_linux_binary.sh tensorflow/lite/micro/tools/make/gen/linux_x86_64/bin/hello_world_test '~~~ALL TESTS PASSED~~~'

results in no output log on the terminal (while the error log is in the /tmp directory):

$ cat /tmp/test_linux_binary/tensorflow/lite/micro/tools/make/gen/linux_x86_64/bin/hello_world_test/logs.txt 
Testing LoadModelAndPerformInference
-0.959f (-1.0*2^-1) near value (-1.0*2^-1) failed at tensorflow/lite/micro/examples/hello_world/hello_world_test.cc:119
0/1 tests passed
~~~SOME TESTS FAILED~~~

The issue is that the -e flag causes the test_linux_binary.sh script to exit before it has a chance to check whether the test passed or failed.

Removing the -e flag fixes the issue.

$ tensorflow/lite/micro/testing/test_linux_binary.sh tensorflow/lite/micro/tools/make/gen/linux_x86_64/bin/hello_world_test '~~~ALL TESTS PASSED~~~'
tensorflow/lite/micro/tools/make/gen/linux_x86_64/bin/hello_world_test: FAIL - '~~~ALL TESTS PASSED~~~' not found in logs.
Testing LoadModelAndPerformInference
-0.959f (-1.0*2^-1) near value (-1.0*2^-1) failed at tensorflow/lite/micro/examples/hello_world/hello_world_test.cc:119
0/1 tests passed
~~~SOME TESTS FAILED~~~

And running the test via the Makefile also works as expected:

$ make -f tensorflow/lite/micro/tools/make/Makefile -j8 test_hello_world_test
tensorflow/lite/micro/tools/make/Makefile:303: warning: overriding recipe for target 'tensorflow/lite/micro/tools/make/downloads/ruy'
tensorflow/lite/micro/tools/make/Makefile:303: warning: ignoring old recipe for target 'tensorflow/lite/micro/tools/make/downloads/ruy'
tensorflow/lite/micro/tools/make/Makefile:303: warning: overriding recipe for target 'tensorflow/lite/micro/tools/make/downloads/person_model_grayscale'
tensorflow/lite/micro/tools/make/Makefile:303: warning: ignoring old recipe for target 'tensorflow/lite/micro/tools/make/downloads/person_model_grayscale'
tensorflow/lite/micro/tools/make/Makefile:303: warning: overriding recipe for target 'tensorflow/lite/micro/tools/make/downloads/person_model_int8'
tensorflow/lite/micro/tools/make/Makefile:303: warning: ignoring old recipe for target 'tensorflow/lite/micro/tools/make/downloads/person_model_int8'
tensorflow/lite/micro/testing/test_linux_binary.sh tensorflow/lite/micro/tools/make/gen/linux_x86_64/bin/hello_world_test '~~~ALL TESTS PASSED~~~'
tensorflow/lite/micro/tools/make/gen/linux_x86_64/bin/hello_world_test: FAIL - '~~~ALL TESTS PASSED~~~' not found in logs.
Testing LoadModelAndPerformInference
-0.959f (-1.0*2^-1) near value (-1.0*2^-1) failed at tensorflow/lite/micro/examples/hello_world/hello_world_test.cc:119
0/1 tests passed
~~~SOME TESTS FAILED~~~

make: *** [tensorflow/lite/micro/examples/hello_world/Makefile.inc:34: test_hello_world_test] Error 1

I have an internal change under review that will fix this issue.