ctest and [!mayfail] tag

[orlandini]

Hello,

Lately I have discovered the [!mayfail] tag for test cases, and I've found it to be quite useful. I am working in a library with several features being developed simultaneously and therefore some tests are currently failing.

In our setup, we have several executables that contain our tests. Running one executable at a time, I get the helpful output from my EventListener::sectionEnded method (EventListener is a class deriving from Catch::EventListenerBase), followed from Catch2's summary, as follows:

The following sections have failed (and were already failing before):
	Failing section 1 (inside a case tagged with [!mayfail])
	Failing section 2 (inside a case tagged with [!mayfail])
	Failing section 3 (inside a case tagged with [!mayfail])
	Failing section 4 (inside a case tagged with [!mayfail])
===============================================================================
test cases:  2 | 1 passed | 1 failed as expected
assertions: 10 | 6 passed | 4 failed as expected

However, when running my tests by running ctest in the build directory of the library, it will simply list that all the tests passed, as ctest will ommit all output from the tests. I have tried registering tests in ctest by the add_test CMake directive and registering individual test cases by the catch_discover_tests function.

Is there a way to list failing test cases? Or at least to allow output from EventListeners when running the whole test suite in ctest?

I am sorry if I didn't make myself clear or if there is something wrong in my understanding of how ctest and Catch2 are related.

[LecrisUT]

Oh I didn't know mayfail exists. Does it return 0 when tests fail? That would explain ehy ctest doesn't report it. You can force these tests to output verbose on the ctest side.

If that works let me know, I have a similar use case

Edit: I've noticed ctest does not support individual test output. I've opened a ticket. In the meantime, is setting ctest to verbose and running the mayfail tests specifically would work? Maybe by making 2 test presets and chaining them in a workflow preset?

[horenmar]

CTest does not have concept of mayfail tests (and even its XFAIL support has bunch of issues), so there is nothing Catch2 can do to integrate them with CTest output.

You can get the full output even when ctest run doesn't fail through --verbose flag, then you will the full output (but the output obviously becomes noisy).