Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak with ndarray as function arg in Cython #3046

Open
ArtificialQualia opened this issue Jul 17, 2019 · 2 comments

Comments

@ArtificialQualia
Copy link

commented Jul 17, 2019

Original issue was created for numpy (numpy/numpy#13237) as I thought this was specific to numpy, but they believe this to be a Cython issue. This issue was uncovered during development of pandas. Below is the issue as opened in the numpy repository:

When using an ndarray as an arg to a cdef function, and that ndarray is not used in the function, that object is never properly garbage collected and causes a memory leak.

If the ndarray is used in the function, it is collected properly.

Reproducing code example:

memleak.pyx

import numpy as np
cimport numpy as cnp
from numpy cimport ndarray
cnp.import_array()

cdef inline void caller():
    cdef:
        ndarray[int] x
    x = np.asarray(range(10000))
    dont_use_x(x)

cdef inline void dont_use_x(ndarray[int] x):
    pass

while True:
    caller()

setup.py:

from distutils.core import setup
from Cython.Build import cythonize
import numpy

setup(
    ext_modules = cythonize("memleak.pyx"),
    include_dirs=[numpy.get_include()]
)

to run:

python setup.py build_ext --inplace
python
import memleak

Numpy/Python version information:

1.16.2 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)]

@scoder

This comment has been minimized.

Copy link
Contributor

commented Jul 18, 2019

Thanks for the report. Sounds like the issue might be Cython's detection of unused variables. It could be that they are handled differently (or not at all) on cleanup. However, the report suggests that buffer / memory-view variables require the usual cleanup here.

@ArtificialQualia

This comment has been minimized.

Copy link
Author

commented Jul 18, 2019

With regards to memoryview, if you replace the line:
cdef inline void dont_use_x(ndarray[int] x):
with:
cdef inline void dont_use_x(long[:] x):
the issue goes away. There is no memory leak.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.