Parallel invocation of doctest's own testsuite via CTest fails

[jktjkt]

I'm on b2611a1, Linux, GCC 8.1.0, CMake 3.12.1, Ninja 1.8.2. After configuration and running a Ninja equivalent of make install, I see that parallel execution of doctest's own test suite leads to many errors.

If these tests require some shared resource, then CMake/CTest configuration should be updated to make that requirement explicit via RESOURCE_LOCK -- see full example at FIXTURES_REQUIRED.

$ ctest -j666
Test project /home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug
      Start 31: executable_dll_and_plugin
      Start  1: main.cpp
      Start 13: coverage_maxout.cpp
      Start 18: list_test_suites
      Start 21: abort_after
      Start 25: filter_3
      Start 19: list_reporters
      Start 10: test_cases_and_suites.cpp
      Start 27: order_2
      Start 23: filter_1
      Start  5: assertion_macros.cpp
      Start 11: asserts_used_outside_of_tests.cpp
      Start 15: help
      Start  3: header.h
      Start  8: logging.cpp
      Start 20: all_binary
      Start  4: alternative_macros.cpp
      Start  6: stringification.cpp
      Start  7: subcases.cpp
      Start  9: templated_test_cases.cpp
      Start 16: count
      Start 26: order_1
      Start 28: order_3
      Start 17: list_test_cases
      Start 24: filter_2
      Start  2: doctest_proxy.h
      Start 22: first_last
      Start 14: version
      Start 29: disabled
      Start 30: exe_with_static_libs
      Start 32: playground
      Start 12: concurrency.cpp
 1/32 Test #31: executable_dll_and_plugin ...........***Failed    0.33 sec
 2/32 Test  #1: main.cpp ............................***Failed    0.33 sec
 3/32 Test #13: coverage_maxout.cpp .................***Failed    0.33 sec
 4/32 Test #18: list_test_suites ....................***Failed    0.33 sec
 5/32 Test #21: abort_after .........................***Failed    0.33 sec
 6/32 Test #25: filter_3 ............................***Failed    0.33 sec
 7/32 Test #19: list_reporters ......................***Failed    0.32 sec
 8/32 Test #10: test_cases_and_suites.cpp ...........***Failed    0.32 sec
 9/32 Test #27: order_2 .............................***Failed    0.30 sec
10/32 Test #23: filter_1 ............................***Failed    0.29 sec
11/32 Test  #5: assertion_macros.cpp ................***Failed    0.28 sec
12/32 Test #11: asserts_used_outside_of_tests.cpp ...***Failed    0.27 sec
13/32 Test #15: help ................................***Failed    0.27 sec
14/32 Test  #3: header.h ............................***Failed    0.24 sec
15/32 Test  #8: logging.cpp .........................***Failed    0.24 sec
16/32 Test #20: all_binary ..........................***Failed    0.23 sec
17/32 Test  #4: alternative_macros.cpp ..............***Failed    0.22 sec
18/32 Test  #6: stringification.cpp .................***Failed    0.22 sec
19/32 Test  #7: subcases.cpp ........................***Failed    0.21 sec
20/32 Test  #9: templated_test_cases.cpp ............***Failed    0.20 sec
21/32 Test #16: count ...............................***Failed    0.19 sec
22/32 Test #26: order_1 .............................***Failed    0.18 sec
23/32 Test #17: list_test_cases .....................***Failed    0.16 sec
24/32 Test #24: filter_2 ............................***Failed    0.15 sec
25/32 Test #28: order_3 .............................***Failed    0.18 sec
26/32 Test #32: playground ..........................   Passed    0.08 sec
27/32 Test #22: first_last ..........................***Failed    0.13 sec
28/32 Test #30: exe_with_static_libs ................   Passed    0.10 sec
29/32 Test  #2: doctest_proxy.h .....................***Failed    0.16 sec
30/32 Test #14: version .............................***Failed    0.14 sec
31/32 Test #29: disabled ............................***Failed    0.12 sec
32/32 Test #12: concurrency.cpp .....................   Passed    0.06 sec

9% tests passed, 29 tests failed out of 32

Total Test time (real) =   0.38 sec

The following tests FAILED:
          1 - main.cpp (Failed)
          2 - doctest_proxy.h (Failed)
          3 - header.h (Failed)
          4 - alternative_macros.cpp (Failed)
          5 - assertion_macros.cpp (Failed)
          6 - stringification.cpp (Failed)
          7 - subcases.cpp (Failed)
          8 - logging.cpp (Failed)
          9 - templated_test_cases.cpp (Failed)
         10 - test_cases_and_suites.cpp (Failed)
         11 - asserts_used_outside_of_tests.cpp (Failed)
         13 - coverage_maxout.cpp (Failed)
         14 - version (Failed)
         15 - help (Failed)
         16 - count (Failed)
         17 - list_test_cases (Failed)
         18 - list_test_suites (Failed)
         19 - list_reporters (Failed)
         20 - all_binary (Failed)
         21 - abort_after (Failed)
         22 - first_last (Failed)
         23 - filter_1 (Failed)
         24 - filter_2 (Failed)
         25 - filter_3 (Failed)
         26 - order_1 (Failed)
         27 - order_2 (Failed)
         28 - order_3 (Failed)
         29 - disabled (Failed)
         31 - executable_dll_and_plugin (Failed)
Errors while running CTest

When run sequentially, I see just one error (#201):

$ ctest 
Test project /home/jkt/work/prog/_build/doctest/GCC_8_1_NETCONF-Debug
      Start  1: main.cpp
 1/32 Test  #1: main.cpp ............................   Passed    0.03 sec
      Start  2: doctest_proxy.h
 2/32 Test  #2: doctest_proxy.h .....................   Passed    0.03 sec
      Start  3: header.h
 3/32 Test  #3: header.h ............................   Passed    0.03 sec
      Start  4: alternative_macros.cpp
 4/32 Test  #4: alternative_macros.cpp ..............   Passed    0.02 sec
      Start  5: assertion_macros.cpp
 5/32 Test  #5: assertion_macros.cpp ................   Passed    0.03 sec
      Start  6: stringification.cpp
 6/32 Test  #6: stringification.cpp .................   Passed    0.02 sec
      Start  7: subcases.cpp
 7/32 Test  #7: subcases.cpp ........................   Passed    0.03 sec
      Start  8: logging.cpp
 8/32 Test  #8: logging.cpp .........................   Passed    0.02 sec
      Start  9: templated_test_cases.cpp
 9/32 Test  #9: templated_test_cases.cpp ............   Passed    0.03 sec
      Start 10: test_cases_and_suites.cpp
10/32 Test #10: test_cases_and_suites.cpp ...........   Passed    0.02 sec
      Start 11: asserts_used_outside_of_tests.cpp
11/32 Test #11: asserts_used_outside_of_tests.cpp ...   Passed    0.03 sec
      Start 12: concurrency.cpp
12/32 Test #12: concurrency.cpp .....................   Passed    0.02 sec
      Start 13: coverage_maxout.cpp
13/32 Test #13: coverage_maxout.cpp .................   Passed    0.03 sec
      Start 14: version
14/32 Test #14: version .............................   Passed    0.03 sec
      Start 15: help
15/32 Test #15: help ................................   Passed    0.02 sec
      Start 16: count
16/32 Test #16: count ...............................   Passed    0.03 sec
      Start 17: list_test_cases
17/32 Test #17: list_test_cases .....................   Passed    0.02 sec
      Start 18: list_test_suites
18/32 Test #18: list_test_suites ....................   Passed    0.03 sec
      Start 19: list_reporters
19/32 Test #19: list_reporters ......................   Passed    0.02 sec
      Start 20: all_binary
20/32 Test #20: all_binary ..........................   Passed    0.03 sec
      Start 21: abort_after
21/32 Test #21: abort_after .........................   Passed    0.02 sec
      Start 22: first_last
22/32 Test #22: first_last ..........................   Passed    0.03 sec
      Start 23: filter_1
23/32 Test #23: filter_1 ............................   Passed    0.02 sec
      Start 24: filter_2
24/32 Test #24: filter_2 ............................   Passed    0.03 sec
      Start 25: filter_3
25/32 Test #25: filter_3 ............................   Passed    0.02 sec
      Start 26: order_1
26/32 Test #26: order_1 .............................   Passed    0.03 sec
      Start 27: order_2
27/32 Test #27: order_2 .............................   Passed    0.02 sec
      Start 28: order_3
28/32 Test #28: order_3 .............................   Passed    0.03 sec
      Start 29: disabled
29/32 Test #29: disabled ............................   Passed    0.02 sec
      Start 30: exe_with_static_libs
30/32 Test #30: exe_with_static_libs ................   Passed    0.03 sec
      Start 31: executable_dll_and_plugin
31/32 Test #31: executable_dll_and_plugin ...........***Failed    0.02 sec
      Start 32: playground
32/32 Test #32: playground ..........................   Passed    0.02 sec

97% tests passed, 1 tests failed out of 32

Total Test time (real) =   0.83 sec

The following tests FAILED:
         31 - executable_dll_and_plugin (Failed)
Errors while running CTest

[onqtam]

Thanks for actually letting me know ctest can run tests in parallel :D
The shared resource in this case is the temp output file from each test, which is then compared with the reference output for that particular test.

I should change it so a different temp output file is used for each test so RESOURCE_LOCK won't be necessary!

[onqtam]

I just pushed a fix in the dev branch

[jktjkt]

Thanks, I can confirm that it works now.