Confusing doctest success/failure

[jcw]

What kind of issue is this?

• PlatformIO Core.
  If you’ve found a bug, please provide an information below.

---

Configuration

MacOS, PIO 6.0.2

Description of problem

A failed test using DocTest reports as follows when running pio test:

================ 1 test cases: 1 failed, 0 succeeded in 00:00:02.240 ================

Whereas it reports this when running pio test -v:

----------------- native:test_test_dummy [PASSED] Took 0.34 seconds -----------------

Steps to Reproduce

1. Set up example in the docs: https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/doctest.html
2. Add a failing test. I added these lines before main():

TEST_CASE("smoke") {
    CHECK(42 == 40 + 1);
}

Additional info

Part of my confusion may also come from the amount of output, which appears to report the failed check twice:

$ pio test
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 1 tests

Processing test_test_dummy in native environment
-------------------------------------------------------------------------------------
Building...
Testing...
test/test_test_dummy/test_main.cpp:7: smoke: CHECK( 42 == 40 + 1 ) is NOT correct!	[FAILED]
----------------- native:test_test_dummy [FAILED] Took 0.35 seconds -----------------

====================================== SUMMARY ======================================
Environment    Test             Status    Duration
-------------  ---------------  --------  ------------
native         test_test_dummy  FAILED    00:00:00.353

______________________________ native:test_test_dummy ______________________________
test/test_test_dummy/test_main.cpp:7:
TEST CASE:  smoke

test/test_test_dummy/test_main.cpp:8: ERROR: CHECK( 42 == 40 + 1 ) is NOT correct!
  values: CHECK( 42 == 41 )

================ 1 test cases: 1 failed, 0 succeeded in 00:00:00.353 ================
$

And here's in verbose mode:

$ pio test -v
Collected 1 tests (test_test_dummy)

Processing test_test_dummy in native environment
-------------------------------------------------------------------------------------
Building...


Testing...
[doctest] doctest version is "2.4.9"
[doctest] run with "--help" for options
===============================================================================
test/test_test_dummy/test_main.cpp:7:
TEST CASE:  smoke

test/test_test_dummy/test_main.cpp:8: ERROR: CHECK( 42 == 40 + 1 ) is NOT correct!
  values: CHECK( 42 == 41 )

===============================================================================
[doctest] test cases: 1 | 0 passed | 1 failed | 0 skipped
[doctest] assertions: 1 | 0 passed | 1 failed |
[doctest] Status: FAILURE!

----------------- native:test_test_dummy [PASSED] Took 0.35 seconds -----------------

====================================== SUMMARY ======================================
Environment    Test             Status    Duration
-------------  ---------------  --------  ------------
native         test_test_dummy  SKIPPED   00:00:00.350
===================== 0 test cases: 0 succeeded in 00:00:00.350 =====================
$

Note: I also have doctest 2.4.9 installed via Homebrew, which appears to override the 2.4.8 installed by PIO.

[ivankravets]

Thanks for the report. Please re-test with pio upgrade --dev.

Does it work now?