Skip to content

Conversation

@guitargeek
Copy link
Contributor

@guitargeek guitargeek commented Jan 15, 2026

All xfail'ed cppyy tests have been reviewed, and the ones that are
actually failing are now marked as xfail(strict=True), so the test
fails if it unexpectedly passes.

This gives us a very useful baseline for what works now in ROOT and what
doesn't, and we'll learn when our developments like the cppyy upgrade on
the CppInterOp-based version will fix some tests.

There are some tests remaining that can't be run because they crash, and
therefore the strict mode doesn't make sense there. But the number of
these tests is not high, and now we always give a reason for why they
crash:

git grep xfail | grep -v "strict"
test_concurrent.py:    @mark.xfail(run=False, reason="Crashes because TClingCallFunc generates wrong code")
test_concurrent.py:    @mark.xfail(run=False, reason="Crashes because the interpreter emits too many warnings")
test_concurrent.py:    @mark.xfail(run=False, reason="segmentation violation")
test_cpp11features.py:    @mark.xfail(run=False, reason = "Crashes")
test_datatypes.py:    @mark.xfail(run=False, reason="segmentation violation")
test_datatypes.py:    @mark.xfail(run=False, reason="error code: Subprocess aborted")
test_doc_features.py:    @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit")
test_fragile.py:    @mark.xfail(run=False, condition=has_asserts(),
test_fragile.py:    @mark.xfail(run=False, reason="Fatal Python error: Aborted")
test_fragile.py:    @mark.xfail(run=False, reason="Fatal Python error: Aborted")
test_fragile.py:    @mark.xfail(run=False, condition=is_modules_off(), reason="Crashes on build with modules off: Fatal Python error: Segmentation fault")
test_lowlevel.py:    @mark.xfail(run=False, condition=IS_WINDOWS, reason="Windows fatal exception: access violation")
test_regression.py:    @mark.xfail(run=WINDOWS_BITS != 64, condition=IS_MAC_ARM | WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit and fails macOS ARM with" \
test_stltypes.py:    @mark.xfail(run=False, reason="Fatal Python error: Segmentation fault")
test_stltypes.py:    @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason="Crashes on Windows 64 bit")
test_templates.py:    @mark.xfail(run=False, reason="error code: Subprocess aborted")

A future development could be to ensure in Cling or cppyy that these
tests at least don't crash but fail gracefully, but that's better to be
done after the cppyy upgrade because some tests might be fixed anyway.

Some tests remain skipped for good reasons:

git grep "mark\.skip"
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:    @mark.skipif(noboost, reason="boost not found")
test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found")
test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found")
test_fragile.py:    @mark.skip(reason="This test is very verbose since it sets gDebug to True")
test_fragile.py:    @mark.skip(reason="Not actually a cppyy test")
test_fragile.py:    @mark.skipif(not has_cpp_20(), reason="std::span requires C++20")
test_leakcheck.py:@mark.skipif(nopsutil == True, reason="module psutil not installed")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_numba.py:@mark.skipif(has_numba == False, reason="numba not found")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:@mark.skipif(has_numba == False, reason="numba not found")
test_eigen.py:    @mark.skipif(eigen_path is None, reason="Eigen not found")
test_pythonify.py:    @mark.skip(reason="Garbage collection tests are fragile")
test_regression.py:    @mark.skip(reason="For ROOT, we don't enable AVX by default ('-mavx' is not passed to Cling)")
test_stltypes.py:@mark.skipif(not has_cpp_20(), reason="std::span requires C++20")

Sorry that these git grep commands don't show the names of the tests!
They are to illustrate how many tests are still marked as skip/xfail
without strict=True.

Closes #20085.

@github-actions
Copy link

github-actions bot commented Jan 16, 2026

Test Results

    22 files      22 suites   3d 11h 21m 54s ⏱️
 3 812 tests  3 812 ✅ 0 💤 0 ❌
76 684 runs  76 684 ✅ 0 💤 0 ❌

Results for commit ae95b68.

♻️ This comment has been updated with latest results.

@guitargeek guitargeek force-pushed the issue-20085 branch 2 times, most recently from e36515c to 16ab12d Compare January 16, 2026 13:01
@guitargeek guitargeek force-pushed the issue-20085 branch 15 times, most recently from 400ec11 to e25b499 Compare January 18, 2026 12:51
@guitargeek guitargeek changed the title [cppyy] Mark most of the xfail-ed tests as "strict" [cppyy] Review xfail'ed tests Jan 18, 2026
@guitargeek guitargeek force-pushed the issue-20085 branch 3 times, most recently from 00e1820 to e81987f Compare January 18, 2026 18:24
All xfail'ed cppyy tests have been reviewed, and the ones that are
actually failing are now marked as `xfail(strict=True)`, so the test
fails if it unexpectedly passes.

This gives us a very useful baseline for what works now in ROOT and what
doesn't, and we'll learn when our developments like the cppyy upgrade on
the CppInterOp-based version will fix some tests.

There are some tests remaining that can't be run because they crash, and
therefore the `strict` mode doesn't make sense there. But the number of
these tests is not high, and now we always give a reason for why they
crash:

```txt
git grep xfail | grep -v "strict"
```
```txt
test_concurrent.py:    @mark.xfail(run=False, reason="Crashes because TClingCallFunc generates wrong code")
test_concurrent.py:    @mark.xfail(run=False, reason="Crashes because the interpreter emits too many warnings")
test_concurrent.py:    @mark.xfail(run=False, reason="segmentation violation")
test_cpp11features.py:    @mark.xfail(run=False, reason = "Crashes")
test_datatypes.py:    @mark.xfail(run=False, reason="segmentation violation")
test_datatypes.py:    @mark.xfail(run=False, reason="error code: Subprocess aborted")
test_doc_features.py:    @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit")
test_fragile.py:    @mark.xfail(run=False, condition=has_asserts(),
test_fragile.py:    @mark.xfail(run=False, reason="Fatal Python error: Aborted")
test_fragile.py:    @mark.xfail(run=False, reason="Fatal Python error: Aborted")
test_fragile.py:    @mark.xfail(run=False, condition=is_modules_off(), reason="Crashes on build with modules off: Fatal Python error: Segmentation fault")
test_lowlevel.py:    @mark.xfail(run=False, condition=IS_WINDOWS, reason="Windows fatal exception: access violation")
test_regression.py:    @mark.xfail(run=WINDOWS_BITS != 64, condition=IS_MAC_ARM | WINDOWS_BITS == 64, reason = "Crashes on Windows 64 bit and fails macOS ARM with" \
test_stltypes.py:    @mark.xfail(run=False, reason="Fatal Python error: Segmentation fault")
test_stltypes.py:    @mark.xfail()
test_stltypes.py:    @mark.xfail(run=False, condition=WINDOWS_BITS == 64, reason="Crashes on Windows 64 bit")
test_templates.py:    @mark.xfail(run=False, reason="error code: Subprocess aborted")
```

A future development could be to ensure in Cling or cppyy that these
tests at least don't crash but fail gracefully, but that's better to be
done after the cppyy upgrade because some tests might be fixed anyway.

Some tests remain skipped for good reasons:
```txt
git grep "mark\.skip"
```
```txt
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:@mark.skipif(noboost == True, reason="boost not found")
test_boost.py:    @mark.skipif(noboost, reason="boost not found")
test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found")
test_eigen.py:@mark.skipif(eigen_path is None, reason="Eigen not found")
test_fragile.py:    @mark.skip(reason="This test is very verbose since it sets gDebug to True")
test_fragile.py:    @mark.skip(reason="Not actually a cppyy test")
test_fragile.py:    @mark.skipif(not has_cpp_20(), reason="std::span requires C++20")
test_leakcheck.py:@mark.skipif(nopsutil == True, reason="module psutil not installed")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_leakcheck.py:    @mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")
test_numba.py:@mark.skipif(has_numba == False, reason="numba not found")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:    @mark.skip(reason="Numba tests comparing execution times are sensitive and fail sporadically.")
test_numba.py:@mark.skipif(has_numba == False, reason="numba not found")
test_eigen.py:    @mark.skipif(eigen_path is None, reason="Eigen not found")
test_pythonify.py:    @mark.skip(reason="Garbage collection tests are fragile")
test_regression.py:    @mark.skip(reason="For ROOT, we don't enable AVX by default ('-mavx' is not passed to Cling)")
test_stltypes.py:@mark.skipif(not has_cpp_20(), reason="std::span requires C++20")
```

Sorry that these `git grep` commands don't show the names of the tests!
They are to illustrate how many tests are still marked as `skip`/`xfail`
without `strict=True`.

Closes root-project#20085.
@guitargeek guitargeek merged commit 50ae29d into root-project:master Jan 19, 2026
29 of 30 checks passed
@guitargeek guitargeek deleted the issue-20085 branch January 19, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[cppyy] Review xfail'ed tests

2 participants