Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix memory leak in reading LC/TPF #1390

Merged
merged 8 commits into from
Dec 5, 2023
Merged

Fix memory leak in reading LC/TPF #1390

merged 8 commits into from
Dec 5, 2023

Conversation

orionlee
Copy link
Collaborator

@orionlee orionlee commented Nov 25, 2023 •
edited

Close #1388

It replaced the workaround deepcopy(hdulist) in PR #1299 . Tests are added to CI memtest workflow to test for memory leaks in read().

Details of changes from PR #1299:

  1. deepcopy(hdulist) in io/generic.py : the codes now close the hdulist, unless it is supplied by the caller.
    The Lightcurve object created has already read copied all the data by the end. So the hdulist is really not needed and can be safely closed.

  2. deepcopy(hdulist) in TargetPixelFile constructor: revert to original behavior.
    In cases that users need to ensure the file handle is closed, they can still call tpf.hdu.close() .

  3. the hdulist.copy() in Lightcurve.hdu property: revert to the original behavior. Here, we give an hdu to user. They should be responsible for managing them.

  4. the hdulist[extension].copy() in TargetPixelFile._open_image(): No change. The shallow copy does work in the specific usage within _open_image() without causing memory leak. Test is changed to test for no unclosed file handle.

  5. The pyproject.toml 's filterwarnings = [ "error::ResourceWarning"] : Revert it. I guess the intent is to make pytest check for unclosed file handles.
    a. specific tests are explicitly marked for testing unclosed file handles instead.
    b. If we do want pytest to check against unclosed file handles in general, "error::pytest.PytestUnraisableExceptionWarning" will also be needed for it to work.
    To do so will require changes in various unrelated tests, e.g., closing HDUList the tests themselves generate.

Changelog

- Fixed memory leak in reading Lightcurve / TargetPixel FITS files [#1390]

@orionlee
Fixed memleak for lc in lightkurve#1388
3244d99 
- revert PR#1299
@orionlee
Fixed memleak for tpf in lightkurve#1388
75824a2 
- revert PR lightkurve#1299
@orionlee
add test for read HDUList
f5c1ec2
@orionlee
Explicit tests for read memory leaks (LC & TPF)
13b57a7 
- Run in memtest workflow in CI (pytest -m memtest --remote-data)
@orionlee
Test tpf.from_fits_images() to ensure no unclosed file handles
584aab9
@orionlee
Revert lc.hdu change in PR lightkurve#1299
cd1c64e
@orionlee
Revert raising ResourceWarning as error during tests in PR lightkurve…
74b4497 
…#1299

- For it to actually work (to ensure no unclosed files), "error::pytest.PytestUnraisableExceptionWarning" wil also be needed
- but it'll create many false alarms.
- Explicit tests on unclosed file handles is done in specific tests instead.
@orionlee
Copy link
Collaborator Author

orionlee commented Nov 28, 2023 •
edited

@christinahedges I intend to merge this PR next week (~Dec 5), if no feedback is given. It fixes the memory leak in v2.4.2 with minimum risk.

@bmorris3 This PR addressed the test failure in lcviz differently by closing the HDUList once its data is copied to the Lightcurve object.

@orionlee
Copy link
Collaborator Author

orionlee commented Dec 4, 2023

@christinahedges I plan to merge the PR fixing the memory leak tomorrow (Dec 5), if there is no feedback.

@orionlee orionlee merged commit 68fdf03 into lightkurve:main Dec 5, 2023
danhey added a commit to danhey/lightkurve that referenced this pull request Dec 13, 2023
@danhey
Squashed commit of the following:
9618e97 
commit 5e4c619
Author: Sam Lee <orionlee@users.noreply.github.com>
Date:   Tue Dec 5 08:21:02 2023 -0800

    pytests: isolate astropy cache from user defaults (lightkurve#1391)

commit eabc909
Author: Sam Lee <orionlee@users.noreply.github.com>
Date:   Tue Dec 5 08:20:24 2023 -0800

    Support QLP changes in sectors 56+ (lightkurve#1392)

    * QLP sector 56+: handle default flux_err column

    * handle QLP-specific quality bitmask

    * docstring updates for QLP sectors 56+

    * add changelog [skip ci]

commit 68fdf03
Author: Sam Lee <orionlee@users.noreply.github.com>
Date:   Tue Dec 5 08:14:18 2023 -0800

    Fix memory leak in reading LC/TPF  (lightkurve#1390)

    * Fixed memleak for lc in lightkurve#1388

    * Fixed memleak for tpf in lightkurve#1388

    * add test for read HDUList

    * Explicit tests for read memory leaks (LC & TPF)
    - Run in memtest workflow in CI (pytest -m memtest --remote-data)

    * Test tpf.from_fits_images() to ensure no unclosed file handles

    * Revert lc.hdu change in PR lightkurve#1299

    * Revert raising ResourceWarning as error during tests in PR lightkurve#1299
    - For it to actually work (to ensure no unclosed files), "error::pytest.PytestUnraisableExceptionWarning" wil also be needed
    - but it'll create many false alarms.
    - Explicit tests on unclosed file handles is done in specific tests instead.

    * add changelog  [skip ci]

commit f8e8c16
Author: Christina Hedges <christina.l.hedges@nasa.gov>
Date:   Fri Nov 3 11:12:48 2023 -0400

    updating to v2.5.0dev [skip ci]

commit 1a6b7c2
Author: Christina Hedges <christina.l.hedges@nasa.gov>
Date:   Fri Nov 3 11:02:06 2023 -0400

    release v2.4.2 [skip ci]

commit 47cbfcf
Author: Christina Hedges <christina.l.hedges@nasa.gov>
Date:   Fri Nov 3 11:01:05 2023 -0400

    releasing v2.4.2

commit e3bd292
Author: Christina Hedges <14965634+christinahedges@users.noreply.github.com>
Date:   Fri Nov 3 01:59:04 2023 -0400

    Revert "Update the stylefile 💅 (lightkurve#1311)" (lightkurve#1382)

    This reverts commit 3b9a0af.

commit 7be1a0f
Author: Christina Hedges <14965634+christinahedges@users.noreply.github.com>
Date:   Thu Nov 2 20:51:37 2023 -0400

    fix changelog and version number

commit 3b9a0af
Author: Daniel <38233719+danhey@users.noreply.github.com>
Date:   Thu Nov 2 10:33:31 2023 -1000

    Update the stylefile 💅 (lightkurve#1311)

    * update stylefile

    * mark edge colors

    * changed lightkurve plotting style

    * update merge conflict [skip ci]

    ---------

    Co-authored-by: Christina Hedges <christina.l.hedges@nasa.gov>

commit 7a7dc2a
Author: Sam Lee <orionlee@users.noreply.github.com>
Date:   Thu Nov 2 12:02:17 2023 -0700

    Fix download error due to no dataURL in MAST result (lightkurve#1380)

    * Fix downlod error due to missing dataURL in MAST result.

    * handle changes in MAST for some Kepler search due to extra KBONUS-BKG
    - some (but not all) Kepler search returns an extra KBONUS-BKG

    * fixing searchresult ordering and HLSP URL

    * update changelog [skip ci]

    ---------

    Co-authored-by: Christina Hedges <christina.l.hedges@nasa.gov>

commit b098e81
Author: Rebekah <rebekahhounsell@gmail.com>
Date:   Thu Nov 2 14:28:28 2023 -0400

    Changed flux_raw to flux_corr for TASOC files (lightkurve#1333)

    * updated to corr

    * trying to re-initiate checks

commit ce610e8
Author: Nschanche <nschanch@umd.edu>
Date:   Thu Nov 2 14:23:19 2023 -0400

    Update searching-for-data-products (lightkurve#1370)

    Modified text to reflect search results

commit dde5582
Author: H. Arda Güler <80536083+arda-guler@users.noreply.github.com>
Date:   Thu Nov 2 21:22:48 2023 +0300

    Remove redundant conditional (lightkurve#1374)

    in regressioncorrector.py

commit 4f2dbed
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Thu Oct 5 10:31:37 2023 -0400

    Bump actions/checkout from 3 to 4 (lightkurve#1367)

    Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
    - [Release notes](https://github.com/actions/checkout/releases)
    - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
    - [Commits](actions/checkout@v3...v4)

    ---
    updated-dependencies:
    - dependency-name: actions/checkout
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 382fd3a
Author: Christina Hedges <christina.l.hedges@nasa.gov>
Date:   Wed Sep 6 09:43:50 2023 -0400

    fix log

commit 8b24061
Author: Nschanche <nicole.e.schanche@nasa.gov>
Date:   Tue Sep 5 17:45:42 2023 -0400

    Updated filename check to fix issue lightkurve#1358 (lightkurve#1364)

commit 5a1d1d1
Author: Christina Hedges <14965634+christinahedges@users.noreply.github.com>
Date:   Tue Sep 5 17:45:01 2023 -0400

    Jupyterhub support (lightkurve#1363)

    * Modify show/interact functions to automatically supply a callable
    for the notebook_url parameter to adapt to operating behind the a
    JupyterHub proxy with randomly generated ports for the Bokeh server.
    ---------

    Co-authored-by: jaytmiller <jmiller@stsci.edu>

commit 6176eb0
Author: Christina Hedges <christina.l.hedges@nasa.gov>
Date:   Tue Sep 5 15:17:26 2023 -0400

    fix numpy bug in search

commit 7d485b6
Author: Zé Vinícius <jvmirca@gmail.com>
Date:   Wed Aug 23 04:42:53 2023 +0800

    Raise a RuntimeError in case arclength cant be computed (lightkurve#1331)

    * check if arclength can be computed, else raise an error

    * handle both quantity and numpy arrays

    ---------

    Co-authored-by: Christina Hedges <christina.l.hedges@nasa.gov>

commit e751c16
Author: Christina Hedges <14965634+christinahedges@users.noreply.github.com>
Date:   Tue Aug 22 16:42:35 2023 -0400

    fixing CDIPs stitching bug (lightkurve#1361)

commit 394246f
Author: Zé Vinícius <jvmirca@gmail.com>
Date:   Wed Aug 23 03:41:00 2023 +0800

    Expose n_iters to the pca method in DesignMatrix (lightkurve#1334)

    * expose n_iters from fbpca so that users can control accuracy of optimality

    * Update src/lightkurve/correctors/designmatrix.py

    Co-authored-by: Dan Foreman-Mackey <dfm@dfm.io>

    * updated changelog

    ---------

    Co-authored-by: Dan Foreman-Mackey <dfm@dfm.io>
    Co-authored-by: Christina Hedges <christina.l.hedges@nasa.gov>

commit edcdd65
Author: Daniel <38233719+danhey@users.noreply.github.com>
Date:   Tue Aug 22 21:13:39 2023 +0200

    fix outlier removal bug (lightkurve#1313)

    * fix outlier removal bug

    * update test

    * add nan to test

    * update comment

    ---------

    Co-authored-by: Christina Hedges <14965634+christinahedges@users.noreply.github.com>

commit 8962d85
Author: Christina Hedges <14965634+christinahedges@users.noreply.github.com>
Date:   Tue Aug 22 15:12:38 2023 -0400

    Updated lightkurve aperture functions to be compliant with numpy 1.25.0 (lightkurve#1360)

    * updated aperture function

    * fixing @Nschanche comments [skip ci]

    * update changelog [skip ci]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Potential memory leak using lightkurve.read
1 participant
@orionlee