astropy.cosmology distance functions using too much memory

[samotracio]

I regulary use distance functions of astropy.cosmology to calculate comoving distances. I noticed they consume large amounts of memory. For a couple million redshifts, my memory usage increases by ~7GB, while 3 million redshifts max out the 16GB system memory.

It seems distances are estimated by quad integration and I know this involves some overhead. But I find it odd to get out of memory error just dealing with a few million distances.

I know I can do this in batches, but shouldn't astropy handle this more elegantly?
Below is a minimal working example.

from astropy.cosmology import LambdaCDM
cosmo = LambdaCDM(H0=100., Om0=0.3, Ode0=0.7)
z = np.random.random(2000000)
dc = cosmo.comoving_distance(z).value

[eteq]

@aconley - any thoughts about this?

[aconley]

We've had other issues like this before. Generally speaking, it's that scipy may not do a good job freeing the memory of it's external c/fortran calls, and that seems to be very dependent on the OS, version of python, version of scipy, etc., etc.

For example, on my laptop (OS X 10.12.6, python 3.6.2, scipy 0.19.1, astropy 2.0.1) that set of commands takes negligible memory (~ 25 MB).

So my advice would be to try updating, and if that doesn't work try to convert it into a scipy replication example and submit a bug to them.

Oh, and if that's really your use case, I would recommend interpolating.

[samotracio]

Thanks for the tip @aconley
It looks indeed the quad routine from scipy was leaking memory until not long ago. Fixed in 0.19.1
For reference, see scipy/scipy#7216

Interpolation would do the trick for my typical usage, but might not be ideal for other users that want to play around with different cosmologies