add support for "expected to fail" tests ?

[fwyzard]

In few places in CMSSW we have unit tests that fail gracefully (exit 0) under some known conditions.
A couple of examples:

• the GPU-based tests fail gracefully if the test machine does not have a supported GPU
• after Triton test fixes #35328, the Triton-based test will not run and fail gracefully on non-amd64 architectures

Maybe we should consider adding a new category of "expected failures" for the tests ?
That is, let tests fail if they need to, but keep a list of tests that are expected to do so, and not consider those as real failures in the IBs/PRs tests.

[fwyzard]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @fwyzard Andrea Bocci.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

@fwyzard , I really do not understand what you are asking for. Instead of adding new category for tests , should not a test decide itself that in which conditions it should fail? If it helps , we can add a startup script e.g. config/SCRAM/runtests.sh --known-exit-codes 1,2,19 -- command opts which still exits with exit code 0 if command fails with known exit codes?

[fwyzard]

@smuzaffar let me give a concrete example.

Given the current test system (a test can only succeed == good or fail == bad) the tests that require CUDA have been written to fail gracefully (exit 0) if CUDA is not available or a GPU is not present.

This means that we have no way of knowing, looking only at the test results, if for a given architecture CUDA is ok and the test ran successfully, or if CUDA is broken and all CUDA-related tests are simply doing exit 0.

I understand that the reason is that e only consider two possible outcomes for each test:

• PASS: it worked;
• FAIL: it failed.

Other projects (e.g. gcc, llvm/clang, etc.) have more possible outcomes for each test:

• PASS: the test was expected to pass, and it passed as expected
• XPASS: the test was expected to fail, but it unexpectedly passed
• FAIL: the test was expected to pass, but it unexpectedly failed
• XFAIL: the test was expected to fail, and it failed as expected

If we had this kind of support, we could let tests fail if the system doesn't meet their requirements (e.g. CUDA is not configured or a GPU is not present), and mark them as XFAIL for that system architecture combination.
Then

• we could mark (even temporarily) tests that are known to fail as XFAIL instead of having to remember about it and ignore those failures
• it could be easier to notice when a test actually fails, instead of silently skipping and exiting with a success,
• we could notice when tests that have been failing for a while actually begin working (e.g. if an architecture gains CUDA support).

[smuzaffar]

for sure I am not getting the point here :-) should not the two outcome enough to cover all 4 of these
PASS: test was expected to pass and it passed OR test was expected to fail and it failed
FAIL: test was expected to pass but it failed OR test was expected to fail but it passed

e.g. following two should be marked as PASS

<test name="AlwaysPass" command="true"/>
<test name="AlwaysFail" command="false" fail="1"/>  #fail="1" is just an example here

and following two should be marked as FAIL

<test name="AlwaysPass" command="false"/>
<test name="AlwaysFail" command="true" fail="1"/>  #fail="1" is just an example here

Now if a test dynamically decides at runtime that it should fail or pass ( e.g depending on the env it is running, availability of GPU or CUDA driver compatibility ) then this should be handled by test itself. For GPU related tests, I would still suggest a startup script/test which decides if a test should pass or fail.