Griddata: "memory leak" (Trac #1869) #2388

Closed
scipy-gitbot opened this Issue Apr 25, 2013 · 13 comments

Comments

Projects
None yet
5 participants
@scipy-gitbot

Original ticket http://projects.scipy.org/scipy/ticket/1869 on 2013-03-18 by trac user MSilva, assigned to @pv.

Runing griddata inside a loop makes the memory usage increase without bound.

Example (from example in the official documentation):

def func(x, y):
return x_(1-x)_np.cos(4_np.pi_x) * np.sin(4_np.pi_y**2)**2

grid_x, grid_y = np.mgrid[0:1:100j, 0:1:200j]
points = np.random.rand(1000, 2)
values = func(points[:,0], points[:,1])

for i in range(100000):

grid_z1 = griddata(points, values, (grid_x, grid_y), method='linear')
@scipy-gitbot

This comment has been minimized.

Show comment Hide comment
@scipy-gitbot

scipy-gitbot Apr 25, 2013

@WarrenWeckesser wrote on 2013-03-19

Perhaps this is because of the Cython bug that Pauli described here: http://projects.scipy.org/scipy/ticket/1868

@WarrenWeckesser wrote on 2013-03-19

Perhaps this is because of the Cython bug that Pauli described here: http://projects.scipy.org/scipy/ticket/1868

@scipy-gitbot

This comment has been minimized.

Show comment Hide comment
@scipy-gitbot

scipy-gitbot Apr 25, 2013

@pv wrote on 2013-03-29

Yes, it's the same. Will be fixed for 0.12.0, and in the future also Cython contains the fix.

@pv wrote on 2013-03-29

Yes, it's the same. Will be fixed for 0.12.0, and in the future also Cython contains the fix.

@orionbelt

This comment has been minimized.

Show comment Hide comment
@orionbelt

orionbelt Nov 27, 2013

I am sorry to report that the bug is still there (or it found a new life, if it was every fixed...)

I am using SciPy 0.13.0 on Gentoo Linux. I also tried it with SciPy 0.12.1, same problem.

I don't know whether it's useful, but the memory leak happens only for 'linear' and 'cubic' methods, not for 'nearest'.

Is there any workaround other than the one suggested here:

http://stackoverflow.com/questions/17438239/memory-leak-in-scipy-griddata-ver-0-12-0

which involves compiling SciPy outside of my distribution's repository? (I did not try it)

Thanks.

I am sorry to report that the bug is still there (or it found a new life, if it was every fixed...)

I am using SciPy 0.13.0 on Gentoo Linux. I also tried it with SciPy 0.12.1, same problem.

I don't know whether it's useful, but the memory leak happens only for 'linear' and 'cubic' methods, not for 'nearest'.

Is there any workaround other than the one suggested here:

http://stackoverflow.com/questions/17438239/memory-leak-in-scipy-griddata-ver-0-12-0

which involves compiling SciPy outside of my distribution's repository? (I did not try it)

Thanks.

@pv

This comment has been minimized.

Show comment Hide comment
@pv

pv Nov 27, 2013

Member

It is fixed. Ask the gentoo people for help.

Member

pv commented Nov 27, 2013

It is fixed. Ask the gentoo people for help.

@jlec

This comment has been minimized.

Show comment Hide comment
@jlec

jlec Nov 29, 2013

Pauli,

It is still reproducible on Opensuse (scipy-0.12.0 / numpy-1.6.2 / cython-0.17.2) and Gentoo (scipy-0.13.1 / numpy-1.8.0 / cython-0.19.2). What else do you need to know for a proper fix?

jlec commented Nov 29, 2013

Pauli,

It is still reproducible on Opensuse (scipy-0.12.0 / numpy-1.6.2 / cython-0.17.2) and Gentoo (scipy-0.13.1 / numpy-1.8.0 / cython-0.19.2). What else do you need to know for a proper fix?

@pv

This comment has been minimized.

Show comment Hide comment
@pv

pv Nov 29, 2013

Member

This issue is fixed in Cython >= 0.18.
EDIT: actually, the fix is only in Cython >= 0.19.

However, Scipy source releases ship with pre-Cythonized .c source files, and you need to remove cythonize.dat before building to ensure they are regenerated. It appears that Scipy 0.13.1 source packages were produced with Cython 0.17.1, which is a buggy version.

@rgommers: apparently, the recent Scipy source packages have been produced with a buggy Cython version (0.17.1). Could you ensure that this issue doesn't reappear in the future? We may also need to consider 0.13.2.

Member

pv commented Nov 29, 2013

This issue is fixed in Cython >= 0.18.
EDIT: actually, the fix is only in Cython >= 0.19.

However, Scipy source releases ship with pre-Cythonized .c source files, and you need to remove cythonize.dat before building to ensure they are regenerated. It appears that Scipy 0.13.1 source packages were produced with Cython 0.17.1, which is a buggy version.

@rgommers: apparently, the recent Scipy source packages have been produced with a buggy Cython version (0.17.1). Could you ensure that this issue doesn't reappear in the future? We may also need to consider 0.13.2.

@pv pv reopened this Nov 29, 2013

@jlec

This comment has been minimized.

Show comment Hide comment
@jlec

jlec Nov 29, 2013

simply removing cythonize.dat does not trigger the rebuilding.

jlec commented Nov 29, 2013

simply removing cythonize.dat does not trigger the rebuilding.

@pv

This comment has been minimized.

Show comment Hide comment
@pv

pv Nov 29, 2013

Member

Indeed, as seen in setup.py.

Member

pv commented Nov 29, 2013

Indeed, as seen in setup.py.

@jlec

This comment has been minimized.

Show comment Hide comment
@jlec

jlec Nov 29, 2013

calling tools/cythonize.py should fix it, right?

jlec commented Nov 29, 2013

calling tools/cythonize.py should fix it, right?

@pv

This comment has been minimized.

Show comment Hide comment
@pv

pv Nov 29, 2013

Member

Yes, or remove both cythonize.dat and PKG-INFO before build.

Member

pv commented Nov 29, 2013

Yes, or remove both cythonize.dat and PKG-INFO before build.

@jlec

This comment has been minimized.

Show comment Hide comment
@jlec

jlec Nov 29, 2013

confirmed. Rebuilding the .c file fixes this issue.

jlec commented Nov 29, 2013

confirmed. Rebuilding the .c file fixes this issue.

@rgommers

This comment has been minimized.

Show comment Hide comment
@rgommers

rgommers Nov 29, 2013

Member

Hmm, that's indeed a good enough reason for a 0.13.2 release. Maybe also include the other ndimage.label regression.

Member

rgommers commented Nov 29, 2013

Hmm, that's indeed a good enough reason for a 0.13.2 release. Maybe also include the other ndimage.label regression.

@pv

This comment has been minimized.

Show comment Hide comment
@pv

pv Feb 24, 2014

Member

This was fixed.

Member

pv commented Feb 24, 2014

This was fixed.

@pv pv closed this Feb 24, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment