Potential memory leak using lightkurve.read

[zclaytor]

Problem description

I am using lightkurve.read to read the Kepler Bonus light curves from my filesystem. The total memory used increases with each new file, even as the variable the light curve is read into is overwritten. After reading a few hundred light curves, not even doing any light curve operations, several GB of memory are used. The behavior persists even when I delete the light curve after reading it in, and when forcing garbage collection.

Example

from glob import glob
import tracemalloc
import lightkurve as lk
# import gc

# files is a list of 4211 KeplerLightCurve fits file paths.
files = sorted(glob("kbonus-bkg/lcs/0033/*/*fits"))
tracemalloc.start()

for i, f in enumerate(files):
    l = lk.read(f)
    # print memory used so far in MB
    a, b = tracemalloc.get_traced_memory()
    print(i, a/1e6, b/1e6)
    # del l
    # gc.collect()

prints the following to the terminal:

0 29.884892 38.138305
1 37.926699 49.275103
2 56.568039 68.800881
3 53.747654 68.800881
4 49.065743 68.800881
5 68.433916 75.703975
6 87.464786 99.720875
7 80.419431 99.720875
8 106.513734 116.28036
9 113.602286 127.28021
10 101.602975 127.28021
11 104.040992 127.28021
12 129.269737 138.434605
13 114.449967 138.434605
14 112.08688 138.434605
15 134.411911 141.053458
...

Expected behavior

I expect the total memory usage to increase for the first few files, but then level off as memory is freed up and reallocated. This is the output when I replace lk.read with astropy.table.Table.read:

0 5.248844 5.645178
1 4.370564 9.53727
2 5.389507 9.749385
3 3.378593 9.749385
4 1.111893 9.749385
5 4.37702 9.749385
6 5.494544 9.827411
7 1.866013 9.827411
8 5.481917 9.827411
9 5.695968 11.061116
10 1.07941 11.061116
11 1.209794 11.061116
12 5.440224 11.061116
13 0.551675 11.061116
14 0.551974 11.061116
15 4.367649 11.061116

Here, the maximum memory used never goes above about 11 GB.

Environment

• platform: Red Hat Enterprise Linux 8.8 (Ootpa)
• lightkurve version: 2.4.2 (installed using conda -c conda-forge) and 2.5.0dev (installed using pip from source in the cloned repository)

[orionlee]

1. I can reproduce problem with TESS SPOC 2 minute lightcurves on Windows, so it is not specific to Kepler Bonus lightcurves / Linux.
2. the problem seems to be a recent regression. The older lightkurve 2.4.0 does not appear to have the memory leak

The problem exists for TESS SPOC lightcurves:

lightkurve 2.5.0dev
astropy 5.0.4
0 8.715249 12.326077
1 11.507044 17.704528
2 13.629003 19.827207
3 17.910356 24.108518
4 18.42802 24.631339
5 21.177316 27.380927
6 25.452262 31.656041
7 26.010972 32.214751
8 28.761221 34.965
9 33.035526 39.239305
10 33.597524 39.801303
...
100 221.514371 227.757565
101 224.226204 230.469398
102 228.460606 234.7038
103 230.504471 236.747665
104 233.216248 239.459442
105 237.450479 243.693673
...
146 318.471775 324.392998
147 318.974317 324.886945
148 321.538435 327.451063
149 324.656307 329.973692
150 325.023099 329.973692

But the slightly older lightkurve v2.4.0 does not have the problem:

lightkurve 2.4.0
astropy 5.0.4
0 6.542636 10.153735
1 7.216337 13.414637
2 7.222926 13.422954
3 9.420149 15.620135
4 7.856869 15.620135
...
119 40.648573 46.893591
120 36.930138 46.893591
121 39.122917 46.893591
...
197 35.755834 46.893591
198 36.766859 46.893591
199 7.098715 46.893591
...

[orionlee]

PR #1299 seems to have introduced the problem.

Workarounds:

1. Downgrade lightkurve to v2.4.0, or
2. if you use the source, you can change the few lines to use the old v2.4.0 logic

For

lightkurve/src/lightkurve/io/generic.py

Lines 37 to 39 in f8e8c16

	else:
	with fits.open(filename) as hdulist:
	hdulist = deepcopy(hdulist)

Replace it with

    else:
        hdulist = fits.open(filename)

Essentially, it revert back to the old logic. The old logic has some issues. But I think they mostly manifest in edge cases.

I don't understand why the new logic cause memory leak though.

[orionlee]

Confirmed that the memory leak indeed stems from the deecopy(hdulist) in the PR.

A sample script that demonstrates the problem.

from glob import glob
import tracemalloc
import astropy
from astropy.io import fits
import lightkurve as lk
from copy import deepcopy

# import gc

# Show lk.read memory leak issues in
# https://github.com/lightkurve/lightkurve/issues/1388

print('lightkurve',  lk.__version__)
print('astropy', astropy.__version__)


def read_lite(filename):
    # simulate the logic in 
    # https://github.com/lightkurve/lightkurve/pull/1299
    hdulist = None
    with fits.open(filename) as hdulist:
        hdulist = deepcopy(hdulist)
    return hdulist

    
# files is a list of sample SPOC TESS lc
files = sorted(glob("spoc_samples/*_lc.fits"))
tracemalloc.start()

for i, f in enumerate(files[:]):
    l = read_lite((f))
    # print memory used so far in MB
    a, b = tracemalloc.get_traced_memory()
    print(i, a / 1e6, b / 1e6)
    # del l
    # gc.collect()

Similar memory leak:

lightkurve 2.5.0dev
astropy 5.0.4
0 2.450405 4.575374
1 4.578252 6.877436
2 6.697511 8.996654
3 8.81539 11.115467
4 10.934333 13.232662
...
145 297.996694 300.220553
146 300.038592 302.264887
147 302.08049 304.306785
148 304.122388 306.348683
149 306.164286 308.390581