BUG: Reading EDF files can be very slow

[skjerns]

I am trying to read an EDF file, but the .read_data() (edit, i mean load_data()) function fails. Same thing happens with this sample file from here

Looking a bit into the code I find that there is a call to _read_segment_file in io\base.py which raises an NotImplementedError. The Problem seems to be the call itself, it doesn't even get to the called functions inside. If I comment the call out, it works fine.

self._read_segment_file(data[:, this_sl], idx, fi,
                                    int(start_file), int(stop_file),
                                    cals, mult)

Strangely enough running the sample file from the EDF-API works without problems

I hope I provided all the info needed. Unfortunately I'm not very experienced in debugging functions or classes. Thanks!

Platform:      Windows-10-10.0.14393-SP0
Python:        3.5.2 (same in 2.7.x) (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)]
Executable:    C:\Program Files\WinPython-3.5\python-3.5.2.amd64\pythonw.exe

mne:           0.14.dev0
numpy:         1.11.2 {lapack=C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include, blas=C:/Program Files (x86)/IntelSWTools/compilers_and_libraries/windows/mkl/include}
scipy:         0.18.1
matplotlib:    1.5.3

sklearn:       0.18
nibabel:       Not found
nitime:        Not found
mayavi:        Not found
nose:          1.3.7
pandas:        0.19.1
pycuda:        Not found
skcuda:        Not found

[jaeilepp]

I'm not sure what you mean by read_data. I can't find that in the code base. The file works fine for me. I read it with mne.io.read_raw_edf('ma0844az_1-1+.edf', preload=True).

_read_segment_file in BaseRaw is an abstract method and the call should direct to this method.

[skjerns]

Sorry, I meant load_data.
In the end preload=True produces the same results for me.
It does not on your system?
I'll have a look later to see where in the _read_segment_file it gets stuck.

[skjerns]

Okay seems like I'm just not being patient enough.
After considerably amount of time the file loaded.

Strange, because files in other formats (brainvision) load much faster (1-2s).
I'm having SDD I/O of ~150MB/s for 15 seconds although the file is only 25MB large.

[larsoner]

I'm having SDD I/O of ~150MB/s for 15 seconds although the file is only 25MB large.

Yikes. That doesn't seem right. Which file are you testing with?

Perhaps there is some optimization we can do. I'll re-title the issue.

[skjerns]

The file linked above (~25MB on disk) takes about 15 seconds to load.

I have another data file (~25MB on disk of sleep recordings ) that is not public and it takes 7.5 minutes to load.

[larsoner]

It takes about 7 seconds on my system, which I agree is too long for a 20MB file (it will not scale well). It should be fixable, we loop in ways we could avoid.

[skjerns]

That would be great. Any idea why my file takes 7.5 minutes despite the same size?

1. file: 1400s @ 250hz @ 38 channels -> 15 seconds
2. file: 30000s @ 100hz @ 5 channels -> 7.5 minutes

[larsoner]

It probably has to do with inefficiencies in how we loop. Different channel/sample configurations will expose those weaknesses differently. Or your other file could require different processing, e.g. resampling if not all channels have the same sample rate, which can be slow.