Use less memory when loading EDF file #10638
Conversation
|
@arnaumanasanch could you please test this PR with your file and report your memory usage? |
|
Thanks @cbrnr The problem seems to be solved with this PR. The current memory usage is the one in the image below.
From a Spike of around 11Gb now it went down to 0.8Gb. This is a size that won't be problematic with the available RAM we have. |
|
Alright, so this is ready for review then! |
|
Done, I'll do the backport after this is merged. |
Co-authored-by: Alexandre Gramfort <alexandre.gramfort@m4x.org>
|
Thanks @cbrnr ! |
* upstream/main: (24 commits) Use less memory when loading EDF file (mne-tools#10638) BUG: fix ipython console accessibility after MNEQtBrowser in Spyder (mne-tools#10637) WIP: Fix mne.time_frequency.multitaper Nyquist adjustment slightly incorrect (mne-tools#10541) BUG: Fix bug with fNIRS reordering (mne-tools#10630) MNT: PyQt6 for pip pre 3.10 (mne-tools#10636) CI: Fix conda (mne-tools#10628) MRG: Update installer links to point to 1.0.3_0 (mne-tools#10622) [BUG, MRG] fix info write access (mne-tools#10626) [MRG] Fixed group_by titles in evoked_plot. (mne-tools#10618) [BUG, MRG] Replace image_interp with interpolation in topomap plot arguments (mne-tools#10617) MAINT: Fix timeout (mne-tools#10624) Simplify mne-tools#10619 (mne-tools#10620) Drop EEG rejection thresholds when replacing EEG with CSD channels (mne-tools#10619) Add get_montage support for fNIRS (mne-tools#10611) ENH: Improve make_head_surface options (mne-tools#10592) [ENH, MRG] Add citation for intracranial JOSS paper (mne-tools#10616) [ENH] Add sys info for mne-icalabel (mne-tools#10615) MRG: Add "highlight" parameter to Evoked.plot() to conveniently highlight time periods (mne-tools#10614) MRG: Allow to pass array of "average" values to plot_evoked_topomap() (mne-tools#10610) fix: snirf length units (mne-tools#10613) ...
|
Is this backporting how-to still accurate? I'm not sure where the manually copied changelog line should go in |
Yes in principle, but in practice if you are not familiar/comfortable with the process it's better to open a PR to maint/1.0 than it is to push directly in the last step
Yes |
|
Thanks @larsoner, I'll try to make a PR then. |
* Use less memory when loading EDF file * Add changelog entry * Formatting Co-authored-by: Alexandre Gramfort <alexandre.gramfort@m4x.org> * Restart CI Co-authored-by: Alexandre Gramfort <alexandre.gramfort@m4x.org>
* Use less memory when loading EDF file (#10638) * Use less memory when loading EDF file * Add changelog entry * Formatting Co-authored-by: Alexandre Gramfort <alexandre.gramfort@m4x.org> * Restart CI Co-authored-by: Alexandre Gramfort <alexandre.gramfort@m4x.org> * Add changelog entry * Fix version * Change version to "unreleased" Co-authored-by: Alexandre Gramfort <alexandre.gramfort@m4x.org>
Fixes #10634. We need to read through the whole file when annotations are present. However, we accidentally prevented garbage collection in a loop, which led to extremely high memory usage. Because we read annotations when
preload=False, this is a problem even when the user does not actually want to load the data. I've fixed this by creating an explicit copy of the subarray in the loop.