[TW#27689] Example unit test project runs no tests (IDFGH-13187)

[detly]

Environment

• Development Kit: Wemos Lolin D32
• Module or chip used: ESP32-WROOM-32
• IDF version (run git describe --tags to find it): v3.3-dev-301-gadd7c49a2
• Build System: CMake
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): 1.22.0-80-g6c4433a
• Operating System: Linux (Ubuntu 18.04)
• Power Supply: USB

Problem Description

There is an example unit test project provided in examples/system/unit_test/test. The docs for it say that it should have output like:

#### Executing one test by its name #####

Running Mean of an empty array is zero...
/Users/ivan/e/esp-idf/examples/system/unit_test/components/testable/test/test_mean.c:16:Mean of an empty array is zero:PASS

-----------------------
1 Tests 0 Failures 0 Ignored 
OK

#### Running tests with [mean] tag #####

Running tests matching '[mean]'...
Running Mean of an empty array is zero...
/Users/ivan/e/esp-idf/examples/system/unit_test/components/testable/test/test_mean.c:16:Mean of an empty array is zero:PASS
Running Mean of a test vector...
/Users/ivan/e/esp-idf/examples/system/unit_test/components/testable/test/test_mean.c:22:Mean of a test vector:PASS
Running Another test case which fails...
/Users/ivan/e/esp-idf/examples/system/unit_test/components/testable/test/test_mean.c:35:Another test case which fails:FAIL: Expected 2147483647 Was -1

-----------------------
3 Tests 1 Failures 0 Ignored 
FAIL

...and so on. However, when I run it on a clean master with idf.py build flash monitor I see:

#### Executing one test by its name #####


-----------------------
0 Tests 0 Failures 0 Ignored 
OK

#### Running tests with [mean] tag #####

Running tests matching '[mean]'...

-----------------------
0 Tests 0 Failures 0 Ignored 
OK

#### Running tests without [fails] tag #####

Running tests NOT matching '[fails]'...

-----------------------
0 Tests 0 Failures 0 Ignored 
OK

#### Running all the registered tests #####


-----------------------
0 Tests 0 Failures 0 Ignored 
OK

#### Starting interactive test menu #####



Press ENTER to see the list of tests.


Here's the test menu, pick your combo:

Enter test for running.

Here's the build log: idf_test_build_log.txt

[detly]

Git bisect (obviously my favourite tool) says 37d30c7:

37d30c7a6e758ee103ae5da9736533086e4581d8 is the first bad commit
commit 37d30c7a6e758ee103ae5da9736533086e4581d8
Author: Renz Christian Bagaporo <renz@espressif.com>
Date:   Sun Nov 11 15:36:10 2018 +0800

    cmake: separate app from idf lib project
    
    mbedtls: import mbedtls using unmodified cmake file

See also #2765 which also has this commit as its possible cause.

[detly]

It actually looks like the unit test code isn't even included in the build tree. If you grep or ack for test_mean.c in the build/ directory before and after that commit, you'll see it's missing (after).

[detly]

Confirmed working again, thanks!

[OOHehir]

For anyone else looking at this..

I experienced the same issue using the VSC plugin, ESP-IDF v5.3-beta2.

I (277) main_task: Started on CPU0
I (277) main_task: Calling app_main()

#### Executing one test by its name #####


#### Running tests with [mean] tag #####


#### Running tests without [fails] tag #####


#### Running all the registered tests #####


#### Starting interactive test menu #####

I managed to solve my issue by setting the icon on the bottom of the IDE, 'ESP-IDF Select port to use' to the appropriate port (it was set correctly already) & then clean, rebuild etc..:

I (282) main_task: Started on CPU0
I (282) main_task: Calling app_main()

#### Executing one test by its name #####

Running Mean of an empty array is zero...
/home/projects/unit_test/components/testable/test/test_mean.c:16:Mean of an empty array is zero:PASS

-----------------------
1 Tests 0 Failures 0 Ignored 
OK

#### Running tests with [mean] tag #####

Running tests matching '[mean]'...
Running Mean of an empty array is zero...
/home/projects/unit_test/components/testable/test/test_mean.c:16:Mean of an empty array is zero:PASS
Running Mean of a test vector...
/home/projects/unit_test/components/testable/test/test_mean.c:22:Mean of a test vector:PASS
Running Another test case which fails...
/home/projects/unit_test/components/testable/test/test_mean.c:35:Another test case which fails:FAIL: Expected 2147483647 Was -1. Function [mean][fails]

-----------------------
3 Tests 1 Failures 0 Ignored 
FAIL

#### Running tests without [fails] tag #####

Running tests NOT matching '[fails]'...
Running Mean of an empty array is zero...
/home/projects/unit_test/components/testable/test/test_mean.c:16:Mean of an empty array is zero:PASS
Running Mean of a test vector...
/home/projects/unit_test/components/testable/test/test_mean.c:22:Mean of a test vector:PASS

-----------------------
2 Tests 0 Failures 0 Ignored 
OK

#### Running all the registered tests #####

Running Mean of an empty array is zero...
/home/projects/unit_test/components/testable/test/test_mean.c:16:Mean of an empty array is zero:PASS
Running Mean of a test vector...
/home/projects/unit_test/components/testable/test/test_mean.c:22:Mean of a test vector:PASS
Running Another test case which fails...
/home/projects/unit_test/components/testable/test/test_mean.c:35:Another test case which fails:FAIL: Expected 2147483647 Was -1. Function [mean][fails]

-----------------------
3 Tests 1 Failures 0 Ignored 
FAIL

#### Starting interactive test menu #####



Press ENTER to see the list of tests.

Done

I don't understand why this impacts the tests running or not but this was the only change between the tests running & not.

Hope this helps others!