fixed excluding an event that happens first and once.

[nafraw]

Reference issue

Fixes #11672.

What does this implement/fix?

Reading EGI file may accidentally excludes an event that happens first and once. This fix made "event.sum() <= 1" as "event.sum() < 1" as the mentioned event above has event.sum() == 1

Additional information

More details are explained in #11672 already.

[welcome]

Hello! 👋 Thanks for opening your first pull request here! ❤️ We will try to get back to you soon. 🚴

[agramfort]

thx @nafraw

can you look into the failing unit tests?

also you'll need to update the latest.inc file to document the bug fix.

thanks

[nafraw]

Hi @agramfort

I updated the latest.inc file. I am not sure if I missed something. There are still errors with the build_doc.

I just noticed that this bug is basically the same as #11626. I don't know if np.count_nonzero(event) is better, but for sure <= 1 is not going to work for the problem I described.

I checked one of the failed test and manually read test_egi.raw using raw=read_raw_egi() as in the unit test with output:
Reading events ...
Assembling measurement info ...
Synthesizing trigger channel "STI 014" ...
Excluding events {CELL, HXX1, SESS, XXY1} ...

Then I use mne.find_events(raw). The output is:
2 events found
Event IDs: [1 2]
array([[19, 0, 1],
[57, 0, 2]], dtype=int64)

It looks fine that events are properly read. Similar to the opinion of #11626, I don't see why the test file should give a warning, ""Did not find any event code with more than one " "event."", The exact issue of this bug is that an event may only happen once and first in some recordings; EGI supports network event, which will be only one sample at the current version (even xml file has a duration tag). The described scenario certainly may happen. A recording with only one event is also technically possible, although rare. As a result, who decides that an event channel has only one event should be excluded must decide what to do or explain if some specific channels with only one event must be excluded. Otherwise, this issue will never end.

[agramfort]

sorry I don't have time to look

[mscheltienne]

For the documentation build, the issue is that you are not listed in names.inc. Could you add yourself?

For the tests, this test is not passing: https://github.com/mne-tools/mne-python/blob/52d1e3e016e5caadb040eb3fa30790a8895d8732/mne/io/egi/tests/test_egi.py#L208C1-L209

But it's not completely clear for me what you are fixing and why this test is now failing. Is it because the fix is wrong or because the test is wrong? Could you share a (small) file with a small code snippet which demonstrate:

• what's the issue when you load the file from main
• how is it fixed when you load the file from this PR

[nafraw]

I added my name and pushed.

This is an example (not super small) file without meaningful EEG data, but may be deleted after some time. You need to unzip with password:
h^^#IrcDnnewt6mH

You can use the codes below to load data. The test_egi.raw is the failed test file.

With the provided file, it has 5 events as can be read from the Events_[...].xml file. Before fixing, MNE ignored the first singular event, 444, leading to only 4 events. After fixing, it returns 5 events. However, for unknown reason and to my understanding, MNE thinks the first and singular* event is not an event and should throw a warning, "Did not find any event code with more than one " "event.". It seems the test file has a singular event and should throw a warning, but now it cannot as this is what I fixed.

How I fixed is clearly described in #11672

*: based on the codes I understand, only happens if an event happens first in the file and only once, since the code is using event.sum() <= 1 to decide. If an event happens only once, but not the first, it is not a problem, as the event code itself is already 2 or larger.

import mne
# egi_fname = r'/mne/io/egi/tests/data/test_egi.raw'
egi_fname = r'/junk_data.mff'
from mne.io import read_raw_egi, read_evokeds_mff, read_raw_fif
raw = read_raw_egi(egi_fname, include=None)
mne.find_events(raw)

[drammock]

I can confirm/reproduce the issue. The check for events.sum() > 1 has been there since the original implementation of the reader; I looked into why the check was there in the first place, and:

1. there are no code comments explaining it
2. I couldn't find any explanation in the original PR [MRG] Egimff #4017, and
3. it's not clear that it was tested very widely before merging

I should add that I haven't personally worked with NetStation hardware so I'm not very familiar with the MFF folder format. This means I can't confidently say whether the original check is sensible or not. I can imagine cases where one might want to exclude an event if and only if it is the first one and it occurs only once (e.g. a standard system-generated event that doesn't reflect experiment design) but I can't tell if that is the logic/motivation here.

There are a few other open issues about failures to read MFF files: #11380, #11188, #11626 (same issue as here), #11627 (draft PR with similar fix as here). Since the changes here seem to be breaking other tests, I've just merged main into #11627 to re-trigger its tests to see if that approach works better (I'm guessing not, since IIUC it's going to exclude more events, not fewer).

@mscheltienne suggested here that we migrate our EGIMFF Raw reader to use mffpy under the hood (which nowadays is already a dependency for reading Epoched MFF files). That seems like the more promising (though harder) approach, and it seems like we should make that a priority. @nafraw @jackz314 @ephathaway @jusjusjus @christianbrodbeck do any of you have time/interest in refactoring our Raw EGI reader to use mffpy?

[larsoner]

Indeed moving to using mffpy is the best long-term solution. I don't have time to work on it currently.

To move the present PR forward, one option would be for someone to use mffpy to extract the events and see if this first one exists in their implementation but not ours, which would be evidence that we shouldn't exclude it.

[drammock]

To move the present PR forward, one option would be for someone to use mffpy to extract the events and see if this first one exists in their implementation but not ours, which would be evidence that we shouldn't exclude it.

I did that and indeed the event is there in mffpy (though they don't read in events by default with their Reader class I don't think... I used mffpy.XML.from_file() which is how they do it in their README)

[jackz314]

Thanks for looking into this! I don't have the capacity to work on this right now but I would be interested in adapting mffpy down the line if I have more time and no one else is working on it.

I think your reasoning for the original check (standard system-generated event) makes sense but I don't think it should be the default IMO. In our case, we had events indicating the start of experiments that were omitted due to this check. For a short-term fix, I think we can disable the check by default, and add the option to explicitly exclude the first one-shot event if people still want the old behavior.