Skip to content

[test] Disable flaky leakcheck tests, as done in ROOT's cppyy - #85

Merged
aaronj0 merged 1 commit into
compiler-research:mainfrom
guitargeek:disable-flaky-leakchecks
Sep 7, 2026
Merged

[test] Disable flaky leakcheck tests, as done in ROOT's cppyy#85
aaronj0 merged 1 commit into
compiler-research:mainfrom
guitargeek:disable-flaky-leakchecks

Conversation

@guitargeek

@guitargeek guitargeek commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

The leak checks in test_leakcheck.py compare RSS before and after a tight call loop, which is unreliable in the presence of arenas and free-lists. The same tests are disabled in ROOT's copy of the cppyy test suite (bindings/pyroot/cppyy/cppyy/test/test_leakcheck.py), so mark them as skipped here with the same reason.

This replaces the macOS-specific xfail marks on test01-test03 and test08, which the unconditional skip supersedes, and with them the now unused IS_MAC and IS_MAC_X86 imports.

test09_initializer_list_argument has no counterpart upstream, as it was added along with the InitializerListConverter fix in #54. It is skipped too: it turned out to be just as sporadic.

@guitargeek

Copy link
Copy Markdown
Collaborator Author

Coincidentally, test09_initializer_list_argument just showed to be flaky in the CI as well:

=================================== FAILURES ===================================
________________ TestLEAKCHECK.test09_initializer_list_argument ________________

self = <test_leakcheck.TestLEAKCHECK object at 0x10bc79f50>

    def test09_initializer_list_argument(self):
        """Leak check of passing a list as an std::initializer_list argument"""
    
        import cppjit
    
        cppjit.cppdef("""\
        namespace LeakCheck {
            int sum_il(std::initializer_list<int> l) {
                int s = 0;
                for (auto i : l) s += i;
                return s;
            }
        }""")
    
        ns = cppjit.gbl.LeakCheck
    
>       self.check_func(ns, "sum_il", [1, 2, 3])

test_leakcheck.py:300: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_leakcheck.TestLEAKCHECK object at 0x10bc79f50>
scope = <namespace cppjit.gbl.LeakCheck at 0x7f91a6259e10>, func = 'sum_il'
args = ([1, 2, 3],), kwds = {}

    def check_func(self, scope, func, *args, **kwds):
        """Leak-check 'func', given args and kwds"""
    
        import gc
    
        # if tmpl_args is provided as a keyword, then this is a templated
        # function that is to be found on each call python-side
        tmpl_args = kwds.pop("tmpl_args", None)
    
        # warmup function
        gc.collect()
        if tmpl_args is None:
            getattr(scope, func)(*args, **kwds)
        else:
            getattr(scope, func)[tmpl_args](*args, **kwds)
    
        # number of iterations
        N = 100000
    
        # The use of arena's, free-lists, etc. means that checking rss remains
        # unreliable, unless looking for consistent jumps, so the leak check will
        # be run M times and only considered failed if it "leaks" every time. In
        # actual practice, the number of fails is 0, 1, or M. Note that the total
        # number of gc objects tracked is always required to remain the same.
        M = 3
    
        # leak check
        fail = 0
        for i in range(M):
            gc.collect()
            pre = len(gc.get_objects())
            last = self.process.memory_info().rss
    
            if tmpl_args is None:
                self.runit(N, scope, func, *args, **kwds)
            else:
                self.runit_template(N, scope, func, tmpl_args, *args, **kwds)
    
            gc.collect()
            assert len(gc.get_objects()) == pre
            if last < self.process.memory_info().rss:
                fail += 1
    
>       assert fail < M
E       assert 3 < 3

test_leakcheck.py:75: AssertionError

I'll disable it too.

@aaronj0

aaronj0 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Coincidentally, test09_initializer_list_argument just showed to be flaky in the CI as well:

Yes this test has failed on the nightlies a couple of times. Although the platform is always intel-mac, I agree that the testing method here is not reliable. There is one more flaky failure when comparing execution times in the numba tests but on very rare occasion.

Comment thread test/test_leakcheck.py Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think all the added skips can be dropped in favour of a single one here

Suggested change
@mark.skip(reason="disabled due to its sporadic nature, especially fragile on VMs")

@guitargeek
guitargeek force-pushed the disable-flaky-leakchecks branch from eb75129 to ca97d14 Compare September 7, 2026 08:01
@guitargeek

Copy link
Copy Markdown
Collaborator Author

Coincidentally, test09_initializer_list_argument just showed to be flaky in the CI as well:

Yes this test has failed on the nightlies a couple of times. Although the platform is always intel-mac, I agree that the testing method here is not reliable. There is one more flaky failure when comparing execution times in the numba tests but on very rare occasion.

Good to know! I prefer to keep this PR just about the leakchecks for now, but let's keep the numba tests in mind for later.

The leak checks in test_leakcheck.py compare RSS before and after a
tight call loop, which is unreliable in the presence of arenas and
free-lists. The same tests are disabled in ROOT's copy of the cppyy
test suite (bindings/pyroot/cppyy/cppyy/test/test_leakcheck.py), so
mark them as skipped here with the same reason.

This replaces the macOS-specific xfail marks on test01-test03 and
test08, which the unconditional skip supersedes, and with them the now
unused IS_MAC and IS_MAC_X86 imports.

test09_initializer_list_argument has no counterpart upstream, as it was
added along with the InitializerListConverter fix in compiler-research#54. It is skipped
too: it turned out to be just as sporadic.
@guitargeek
guitargeek force-pushed the disable-flaky-leakchecks branch from ca97d14 to 7e323db Compare September 7, 2026 08:11
@aaronj0

aaronj0 commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

ruff errors are unrelated, fixed in #84

@aaronj0 aaronj0 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@aaronj0
aaronj0 merged commit 928e2e7 into compiler-research:main Sep 7, 2026
25 of 26 checks passed
@guitargeek
guitargeek deleted the disable-flaky-leakchecks branch September 7, 2026 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants