I'm testing gevent with Cython 3. gevent has a combination of Cython code in .pyx/pxd and pure-Python code annotated and compiled with .pxd files. (Which is awesome, BTW. Thanks again for a great project!)
gevent's test suite has a job that runs all the tests and watches for memory leaks. It basically does this by using gc.get_objects() to count all the objects extent, run a particular test, and count the objects again. If the counts don't match, this process is repeated a few times to see if a pattern emerges. When it's been run enough times to see that a particular type of object is always gaining new instances after a test run, the test is declared to leak and is failed.
Using Cython 0.29.17, this test job passes.
The test job with Cython 3.0a4 fails; many tests report leaks. For example:
FAIL: test_raises_timeout_Timeout_exc_customized (__main__.TestGet)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 214, in wrapper
return _RefCountChecker(self, method)(args, kwargs)
File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 192, in __call__
while self._check_deltas(growth):
File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 168, in _check_deltas
raise AssertionError('refcount increased by %r\n%s' % (deltas, diff))
AssertionError: refcount increased by [9, 1168, 2, 2, 2]
RuntimeError 6 +1
Timeout 10 +1
That's showing that the last three runs of this test resulted in two new objects each time; one new RuntimeError and one new Timeout (a kind of exception). I thought at first that maybe it was just exceptions leaking since the first reports are just of exceptions, but later on we see a variety of objects leaking:
840 FAIL: test_unordered (__main__.TestErrorInIterator)
841 ----------------------------------------------------------------------
842 Traceback (most recent call last):
843 File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 214, in wrapper
844 return _RefCountChecker(self, method)(args, kwargs)
845 File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 192, in __call__
846 while self._check_deltas(growth):
847 File "/home/travis/.runtimes/snakepit/2.7.d/lib/python2.7/site-packages/gevent/testing/leakcheck.py", line 168, in _check_deltas
848 raise AssertionError('refcount increased by %r\n%s' % (deltas, diff))
849 AssertionError: refcount increased by [17, 1187, 10, 10, 10]
850 list 2061 +2
851 deque 5 +1
852 ThreadPool 5 +1
853 instancemethod 116 +1
854 set 291 +1
855 Queue 5 +1
856 Semaphore 5 +1
857 fork 5 +1
858 _Condition 7 +1
I've been using language_level=3str in most places for some time already. The only other thing that caught my eye as a possible cause was the change to make binding True by default. But I changed that directive in my setup.py and tested locally and still get the failures.
Locally I've tested 3.0a1 through 3.0a4, inclusive, and get the failures on all versions. (I also tried git master as-of now—I think—and have the same issues.)
Next, I tried not using Cython compiled versions of the pure-Python modules, keeping just the compiled .pyx files, and locally the tests pass again (let me know if that wasn't clear). So it appears to have something to do with that mode?
I haven't tried to create a smaller example yet because I was hoping the problem might just be obvious :) Please let me know what other info I can provide or how I can help track this down.
I'm testing gevent with Cython 3. gevent has a combination of Cython code in .pyx/pxd and pure-Python code annotated and compiled with .pxd files. (Which is awesome, BTW. Thanks again for a great project!)
gevent's test suite has a job that runs all the tests and watches for memory leaks. It basically does this by using
gc.get_objects()to count all the objects extent, run a particular test, and count the objects again. If the counts don't match, this process is repeated a few times to see if a pattern emerges. When it's been run enough times to see that a particular type of object is always gaining new instances after a test run, the test is declared to leak and is failed.Using Cython 0.29.17, this test job passes.
The test job with Cython 3.0a4 fails; many tests report leaks. For example:
That's showing that the last three runs of this test resulted in two new objects each time; one new
RuntimeErrorand one newTimeout(a kind of exception). I thought at first that maybe it was just exceptions leaking since the first reports are just of exceptions, but later on we see a variety of objects leaking:I've been using
language_level=3strin most places for some time already. The only other thing that caught my eye as a possible cause was the change to makebindingTrue by default. But I changed that directive in mysetup.pyand tested locally and still get the failures.Locally I've tested 3.0a1 through 3.0a4, inclusive, and get the failures on all versions. (I also tried git master as-of now—I think—and have the same issues.)
Next, I tried not using Cython compiled versions of the pure-Python modules, keeping just the compiled .pyx files, and locally the tests pass again (let me know if that wasn't clear). So it appears to have something to do with that mode?
I haven't tried to create a smaller example yet because I was hoping the problem might just be obvious :) Please let me know what other info I can provide or how I can help track this down.