MRG eeglab .set reader #2676

jasmainak · 2015-12-06T23:01:47Z

very much WIP and works only for the sample data provided by EEGLAB at the moment. If anyone has files to share, I'd be happy to try them.

TODOs

Fix scaling issue when plotting raw
Check if chanlocs already exists in the set file
Check if data already exists in the set file
Handle epochs data
Make it work with .dat file
eog topoplot

agramfort · 2015-12-07T08:43:51Z

doc/python_reference.rst

@@ -107,6 +107,7 @@ Functions:
  read_raw_edf
  read_raw_kit
  read_raw_nicolet
+  read_raw_set


read_raw_eeglab ?

jasmainak · 2015-12-07T13:28:14Z

okay guys, we have preload option now :) Thanks to @jaeilepp for debugging help. It works when data is in a separate fdt file. I have to think how we can handle preload when it's in the set file itself.

mmagnuski · 2015-12-07T13:29:42Z

mne/io/eeglab/eeglab.py

+        # get info
+        if ch_fname is not None:
+            ch_fname = op.join(basedir, ch_fname)
+        info = _get_info(eeg, eog, ch_fname)


Many set files already contain channel information. You can check eeg.chanlocs to see if it is true.
eeg.chanlocs is a matlab structure with following fields: labels, Y, X, Z, sph_theta, sph_phi, sph_radius, theta, radius, type, ref, urchan. If channel positions are present - all coordinate fields should be filled. labels contains channel names.
For example in matlab:

EEG.chanlocs(5)

gives:

ans = labels: 'E5' Y: -3.0769 X: 10.1424 Z: 1.2423 sph_theta: -16.8763 sph_phi: 6.6854 sph_radius: 10.6714 theta: 16.8763 radius: 0.4629 type: '' ref: 'average' urchan: 5

Maybe you have an example file?

Yes, here is a Dropbox link:
https://www.dropbox.com/sh/4jl2bki7o6rqtmt/AAACrmfO8VtfqQc9mEjtIDIoa?dl=0

cool, I'll check

larsoner · 2015-12-07T14:08:17Z

mne/io/eeglab/eeglab.py

+    -------
+    raw : Instance of RawSet
+        A Raw object containing EEGLAB .set data.
+    """


add Notes section with versionadded directive

larsoner · 2015-12-07T14:13:53Z

Those files add ~4MB to the repo. I think I'd rather put them in the testing dataset to avoid repo bloat. @agramfort WDYT? What's the limit on adding files?

larsoner · 2015-12-07T14:17:18Z

@jasmainak there is no way to get segments of data using loadmat, right? I guess if data were in MATLAB's newer HDF5 .mat files we could. But since we probably can't, I think for preload=False with data in the set file you can just throw a warning if preload is not True (since memmapping probably won't be easy/worth it either) and say that it will be forced to True since data is in the set file.

larsoner · 2015-12-07T14:18:35Z

mne/io/eeglab/tests/test_eeglab.py

+def test_io_set():
+    """Test importing EEGLAB .set files"""
+    _test_raw_reader(read_raw_set, True, fname=fname,
+                     ch_fname=ch_fname)


Don't forget to test degenerate cases like not ch_fname.endswith('.locs') as well

jasmainak · 2015-12-07T15:01:30Z

@Eric89GXL I was thinking maybe we could use io.whosemat to query if the eeg.data attribute is a string or an array and then try to do an fseek to that location. Wdyt?

jasmainak · 2015-12-07T15:04:48Z

@Eric89GXL yeah the files are large at the moment. We can replace them with smaller files when we are closer to merging. That is why it's a separate commit. I'll overwrite the commit.

larsoner · 2015-12-07T15:14:00Z

We could fseek and read if you can figure out how the data are actually stored in the .mat file. That would be cool.

jasmainak · 2015-12-23T10:53:48Z

@agramfort comments addressed

agramfort · 2015-12-23T13:30:35Z

merged by rebase after my cosmit to force upper case.

agramfort · 2015-12-23T13:30:40Z

thanks heaps @jasmainak !

jona-sassenhagen · 2015-12-23T13:35:30Z

Wanna make an announcement on the EEGLAB mailing list? :)

agramfort · 2015-12-23T13:38:51Z

Wanna make an announcement on the EEGLAB mailing list? :)

I am responsible for this ...

jasmainak · 2015-12-23T15:54:59Z

awesome, looks like I missed a lot of action in a few hours AFK :) Thanks for the reviews

teonbrooks · 2015-12-23T16:51:12Z

thanks @jasmainak! this is awesome. i know plenty of colleagues who would love to hear about this

jona-sassenhagen · 2015-12-24T02:46:13Z

Okay this is possibly a stupid question, but how do I access events for a raw file? From reading the docs or code, I can't see where that would be handled, and there is no stim channel in the resulting object.

jasmainak · 2015-12-24T04:33:37Z

I think you will have to use raw.set_channel_type to set the stim channel manually.

I don't think you can have raw.events. I think the cleanest solution is to handle it with this issue #2665 which many people are looking for too.

In the meanwhile. maybe you could artificially create a stim channel (if there isn't one) from the events in Matlab and add it to the data?

agramfort · 2015-12-24T09:13:49Z

if the events are in the .set files we should add a stim_channel automatically. That's how it's done in edf for example. are events in .set files?

jasmainak · 2015-12-24T10:00:39Z

yeah, I think they will be there in the .set file. The problem however is that you don't always know the duration of the events. So, generating the stim channel from the events is an ill posed problem ;) I think the events annotation we have discussed is the correct way to go about it. It will solve many problems at once.

jona-sassenhagen · 2015-12-24T10:24:10Z

In the brainvision reader, if we don't know the length, we just set them to zero. The problem is rather that EEGLAB has much richer event handling, including arbitrary strings as names.

(Stimulus channels are not the EEGLAB way. Something I have to add to my EEGLAB switcher docs. @jasmainak where should I put that file btw?)

jasmainak · 2015-12-24T10:42:58Z

@jona-sassenhagen feel free to share it by email or FB message as you've done previously.

jona-sassenhagen · 2015-12-24T10:50:13Z

I'll try to make a PR asap. Just tell me what file to put it into.

jasmainak · 2015-12-24T11:06:35Z

ah, sorry I misunderstood your question. I think it should go to the manual. A file in manual/migrating.rst? Makes sense?

teonbrooks · 2015-12-24T16:29:48Z

i agree, there should be an auto detection option for the trigger, especially if this is in the release. all the readers have one, even it's a simplified version. as @jona-sassenhagen mentioned, if the duration is unknown we give it a duration of a sample.

jona-sassenhagen · 2015-12-24T16:32:58Z

I already made a PR Teon :) I'm not sure how to create a trigger channel though, I'm too stupid to figure out that part of the BV code.

teonbrooks · 2015-12-24T16:49:44Z

follow up to this pr: #2745

I'll join in over there

jasmainak force-pushed the eeglab_io branch 2 times, most recently from 2f5bced to c46bb00 Compare December 7, 2015 00:27

agramfort reviewed Dec 7, 2015
View reviewed changes

doc/python_reference.rst

@@ -107,6 +107,7 @@ Functions:

read_raw_edf

read_raw_kit

read_raw_nicolet

read_raw_set

Copy link

Member

agramfort Dec 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

read_raw_eeglab ?

jasmainak force-pushed the eeglab_io branch from e46bc67 to ffad6f2 Compare December 7, 2015 13:29

mmagnuski reviewed Dec 7, 2015
View reviewed changes

larsoner mentioned this pull request Dec 7, 2015

MRG: Allow preload=False for readers #2668

Merged

larsoner reviewed Dec 7, 2015
View reviewed changes

jasmainak force-pushed the eeglab_io branch 3 times, most recently from 80dac1c to 1a05244 Compare December 7, 2015 16:49

jasmainak added 9 commits December 23, 2015 16:17

FIX raise error if .dat format data is present

67ea324

TST: Home run, last minute tweak

223894f

Address Eric's comments

29f2f28

FIX for rescaling problem

0a70015

Minor fixes

d463a73

FIX avg refs

f98a4d2

Hack for eeg proj not needed any more

3c14d34

FIX address comments

52a3da2

address comments by alex

996409a

jasmainak force-pushed the eeglab_io branch from 7b55a02 to 996409a Compare December 23, 2015 10:47

agramfort merged commit 996409a into mne-tools:master Dec 23, 2015

jasmainak deleted the eeglab_io branch December 24, 2015 10:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MRG eeglab .set reader #2676

MRG eeglab .set reader #2676

jasmainak commented Dec 6, 2015

agramfort Dec 7, 2015

jasmainak commented Dec 7, 2015

mmagnuski Dec 7, 2015

jona-sassenhagen Dec 7, 2015

mmagnuski Dec 7, 2015

jasmainak Dec 7, 2015

larsoner Dec 7, 2015

larsoner commented Dec 7, 2015

larsoner commented Dec 7, 2015

larsoner Dec 7, 2015

jasmainak commented Dec 7, 2015

jasmainak commented Dec 7, 2015

larsoner commented Dec 7, 2015

jasmainak commented Dec 23, 2015

agramfort commented Dec 23, 2015

agramfort commented Dec 23, 2015

jona-sassenhagen commented Dec 23, 2015

agramfort commented Dec 23, 2015

jasmainak commented Dec 23, 2015

teonbrooks commented Dec 23, 2015

jona-sassenhagen commented Dec 24, 2015

jasmainak commented Dec 24, 2015

agramfort commented Dec 24, 2015 via email

jasmainak commented Dec 24, 2015

jona-sassenhagen commented Dec 24, 2015

jasmainak commented Dec 24, 2015

jona-sassenhagen commented Dec 24, 2015

jasmainak commented Dec 24, 2015

teonbrooks commented Dec 24, 2015

jona-sassenhagen commented Dec 24, 2015

teonbrooks commented Dec 24, 2015

MRG eeglab .set reader #2676

MRG eeglab .set reader #2676

Conversation

jasmainak commented Dec 6, 2015

agramfort Dec 7, 2015

Choose a reason for hiding this comment

jasmainak commented Dec 7, 2015

mmagnuski Dec 7, 2015

Choose a reason for hiding this comment

jona-sassenhagen Dec 7, 2015

Choose a reason for hiding this comment

mmagnuski Dec 7, 2015

Choose a reason for hiding this comment

jasmainak Dec 7, 2015

Choose a reason for hiding this comment

larsoner Dec 7, 2015

Choose a reason for hiding this comment

larsoner commented Dec 7, 2015

larsoner commented Dec 7, 2015

larsoner Dec 7, 2015

Choose a reason for hiding this comment

jasmainak commented Dec 7, 2015

jasmainak commented Dec 7, 2015

larsoner commented Dec 7, 2015

jasmainak commented Dec 23, 2015

agramfort commented Dec 23, 2015

agramfort commented Dec 23, 2015

jona-sassenhagen commented Dec 23, 2015

agramfort commented Dec 23, 2015

jasmainak commented Dec 23, 2015

teonbrooks commented Dec 23, 2015

jona-sassenhagen commented Dec 24, 2015

jasmainak commented Dec 24, 2015

agramfort commented Dec 24, 2015 via email

jasmainak commented Dec 24, 2015

jona-sassenhagen commented Dec 24, 2015

jasmainak commented Dec 24, 2015

jona-sassenhagen commented Dec 24, 2015

jasmainak commented Dec 24, 2015

teonbrooks commented Dec 24, 2015

jona-sassenhagen commented Dec 24, 2015

teonbrooks commented Dec 24, 2015