Drastically reduce memory usage during epoching

[hoechenberger]

Peak memory consumption during epoching reduced by more than 50%!
And we're also faster now!

Benchmarks

Epochs creation during make_epochs script

main

this branch

Epochs creation during run_ica script

main

this branch

How this was achieved

• We preload epochs so we can remove the respective raw from memory
• Instead of creating a list of all epochs for all runs before concatenation and then concatenating them all at once, we now concatenate epochs immediately after they were created. This further reduced peak memory usage significantly (although I don't fully understand why!)
• During ICA epochs generation, we now drop all epochs that are not in config.conditions. In main, we create epochs from ALL events, often creating uselessly huge numbers of epochs, increasing memory footprint and massively slowing down ICA decomposition.
• We now only keep a max of 300 EOG and ECG epochs (hardcoded limit)

Further ideas

We may be able to further reduce peak memory consumption at the cost of I/O and processing speed.

• For example, currently we're creating EOG and ECG epochs in the same loop as the "task" epochs. If we split this into separate loops, peak memory consumption could be lower.
• Currently, EOG and ECG epochs are reduced to 300 elements only AFTER the bloated epochs objects have been created. Instead, one could first only extract EOG and ECG events, subset those to get 300, and only then create the epochs.

Before merging …

• Changelog has been updated (docs/source/changes.md)

[hoechenberger]

I just disabled EOG and ECG epochs creation during ICA and this is what the mprof looks like then:

So this is definitely a must-investigate.

[hoechenberger]

Current status during the ICA script, after implementing the changes we discussed during our Zoom call @agramfort

There's still room for improvement but I guess for now it's okay to move on!

[hoechenberger]

Ah, good catch! A true mathematician 😁

[hoechenberger]

Merci @agramfort!