Disk caching of preprocessing/transformation result

[PierreGtch]

As mentioned in #367, it would be great to have the option to save on disk the results of computationally expensive preprocessing/transformations.

Such disk cache result should be unique for every combination of:

• dataset code
• dataset args
• paradigm name
• paradigm args
• eventually transformer (see Allow passing fixed transformers to evaluations #367)

Bonus: save the preprocessed data in a BIDS format!

[PierreGtch]

It could be interesting to introduce a notion of paradigm hierarchy, for example the following would evaluate to true:

MotorImagery(events=['right_hand', 'left_hand']) <= MotorImagery(events=['right_hand', 'left_hand', 'feet']) 
# True

FilterBankMotorImagery(filters=[(8, 12), (12, 16)]) <= MotorImagery(fmin=1, fmax=40) 
# True (but we could have edge effects if we apply filters on epochs directly)

MotorImagery(channels=["C3",]) <= MotorImagery(channels=None) 
# tricky... a dataset can be without channel C3, even if we use all it's channels

The semantic of a<=b would be: a can be computed from b.
This way, for every new preprocessing we want to compute, if the result of a preprocessing higher in the hierarchy has already been cached, we could re-use it instead of loading again the raw signals.

[sylvchev]

Interesting. Let's discuss this in the BCI meeting.

[PierreGtch]

Yes looking forward to it!

[PierreGtch]

Another note: maybe we should save the preprocessed raws on disk because the expensive steps of the pre-processings are loading the data, applying the frequency filters, and resampling.
If we saved the raw data (already filtered and resampled), we could then read it with preload=False and the epoching would only load in memory the channels and events we need.

With this solution, each cached dataset would use more disk space but they would also be more general. Also, the BIDS format is only compatible with mne.Raw, not mne.Epochs (https://mne.tools/mne-bids/stable/index.html#supported-file-formats)

[sylvchev]

Ok, this is something to consider as preloading is mandatory for MOABB but it is a big limitation for huge datasets (like those that could use to train DL). Also, if we could have some clever approach that encompass BIDS format, this will really be nice, see #391