Support writing custom event durations

[cbrnr]

Fixes #30. I have written some custom functions to convert mne.Annotations into a suitable (n, 3) ndarray - the question is where should I include these functions? Maybe the best place would be directly in MNE. Here's the code:

def annotations_to_array(annotations, fs):
    array = np.zeros((len(annotations), 3), dtype=int)
    ids = _unique_ids(annotations.description)
    for row, annotation in enumerate(annotations):
        array[row, 0] = annotation["onset"] * fs
        array[row, 1] = ids[annotation["description"]]
        array[row, 2] = annotation["duration"] * fs
    return array

def _unique_ids(ids):
    ids = {id: _str_to_id(id) for id in ids}
    for k, v in ids.items():
        if v is None:
            ids[k] = _unused_id(ids)
    return ids


def _unused_id(ids):
    id = 0
    while id in ids.values():
        id += 1
    return id

def _str_to_id(s):
   number = "".join([c for c in s if c.isdigit()])
   if number:
       return int(number)
   else:
       return None

So if you want to convert existing annotations, you call events = annotations_to_array(raw.annotations, raw.info["sfreq"]). This gives something very similar to what you get with mne.events_from_annotations, but with a valid duration column and better(TM) mapping of descriptions to event IDs.

Example using this dataset:

import mne

raw = mne.io.read_raw_edf("S001R04.edf", preload=True)
events1 = annotations_to_array(raw.annotations, raw.info["sfreq"])
events2 = mne.events_from_annotations(raw)

Original raw.annotations.description:

array(['T0', 'T2', 'T0', 'T1', 'T0', 'T1', 'T0', 'T2', 'T0', 'T2', 'T0',
       'T1', 'T0', 'T2', 'T0', 'T1', 'T0', 'T2', 'T0', 'T1', 'T0', 'T1',
       'T0', 'T2', 'T0', 'T1', 'T0', 'T2', 'T0', 'T1'], dtype='<U2')

events1 (note that the numbers match the numbers found in 'T0', 'T1', and 'T2', plus there is a valid durations column):

array([[    0,     0,   672],
       [  672,     2,   656],
       [ 1328,     0,   672],
       [ 2000,     1,   656],
       [ 2656,     0,   672],
       [ 3328,     1,   656],
       [ 3984,     0,   672],
       [ 4656,     2,   656],
       [ 5312,     0,   672],
       [ 5984,     2,   656],
       [ 6640,     0,   672],
       [ 7312,     1,   656],
       [ 7968,     0,   672],
       [ 8640,     2,   656],
       [ 9296,     0,   672],
       [ 9968,     1,   656],
       [10624,     0,   672],
       [11296,     2,   656],
       [11952,     0,   672],
       [12624,     1,   656],
       [13280,     0,   672],
       [13952,     1,   656],
       [14608,     0,   672],
       [15280,     2,   656],
       [15936,     0,   672],
       [16608,     1,   656],
       [17264,     0,   672],
       [17936,     2,   656],
       [18592,     0,   672],
       [19264,     1,   656]])

events2 (note that event IDs start at 1 so they don't match 'T0', 'T1', and 'T2', plus the second column is redundant and doesn't contain any information):

(array([[    0,     0,     1],
        [  672,     0,     3],
        [ 1328,     0,     1],
        [ 2000,     0,     2],
        [ 2656,     0,     1],
        [ 3328,     0,     2],
        [ 3984,     0,     1],
        [ 4656,     0,     3],
        [ 5312,     0,     1],
        [ 5984,     0,     3],
        [ 6640,     0,     1],
        [ 7312,     0,     2],
        [ 7968,     0,     1],
        [ 8640,     0,     3],
        [ 9296,     0,     1],
        [ 9968,     0,     2],
        [10624,     0,     1],
        [11296,     0,     3],
        [11952,     0,     1],
        [12624,     0,     2],
        [13280,     0,     1],
        [13952,     0,     2],
        [14608,     0,     1],
        [15280,     0,     3],
        [15936,     0,     1],
        [16608,     0,     2],
        [17264,     0,     1],
        [17936,     0,     3],
        [18592,     0,     1],
        [19264,     0,     2]]), {'T0': 1, 'T1': 2, 'T2': 3})

Of course, annotations_to_array could also return the mapping just like mne.events_from_annotations does.

WDYT (also including @agramfort, @massich, and @larsoner regarding the inclusion of these functions into MNE)?

[agramfort]

I don't see any use of such a public function in mne but I may be wrong.

[cbrnr]

We could change the behavior of mne.events_from_annotations so that the second column contains durations instead of all zeros.

[agramfort]

it would not be a MNE event array then. The second column of the MNE events tell you what was the previous value of the stim channel channel. So events and the stim channel capture the same information. This may be clarifier in https://mne.tools/dev/glossary.html event paragraph.

[cbrnr]

Yes, I know that this won't be an event array then, but currently we lose information when using events instead of annotations (namely the durations). But I understand that this might not fit into MNE, unless we want to facilitate exporting data to BrainVision.

[cbrnr]

I could include these functions in MNELAB, but pybv currently has the problem that it cannot perform a round-trip reading/writing data conversion without the ability to write durations (the conversion of custom annotation descriptions to event IDs is another problem, but at least this information is just converted and not lost).

[cbrnr]

Another idea: maybe mne.events_from_annotations could output a third element in the tuple (in addition to the events array and event type mapping), namely an array with event durations?

[sappelhoff]

Another idea: maybe mne.events_from_annotations could output a third element in the tuple (in addition to the events array and event type mapping), namely an array with event durations?

This seems unobtrusive to me 👍 and it would be a nice addition to have

[larsoner]

+1 for return_durations kwarg and optional return

[agramfort]

ok with me

…

[massich]

I remember to have had a discussion about the durations. (with pretty much the same conclusions).

Apart from that, I'm not sure that with the current API you need the private functions of your snippet. You should be able to unpack it and do the very same at once:

event_1 = np.stack(raw.samples(annot.onset),
                   annot.info['sfreq'] * annot.duration,  #raw.samples(annot.duration) is abusing but it should also work
                   some_maping(annot.description))

But I need look at it better.

[massich]

Then to me, the right move would it be to be able to construct epochs from annotations directly. Without this events_from_annotations thing which seems unnatural.

[cbrnr]

I agree, directly supporting annotations would be the way to go. The best solution would be to add support for exporting to BV directly in MNE. However, we don't have an API for that yet because there's only raw.save, which only saves to FIF. I think this needs a bit of discussion, so the rather ugly workaround using mne.events_from_annotations might be the way to go for now.

[codecov]

Codecov Report

Merging #33 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #33      +/-   ##
==========================================
+ Coverage   95.39%   95.45%   +0.06%     
==========================================
  Files           3        3              
  Lines         217      220       +3     
==========================================
+ Hits          207      210       +3     
  Misses         10       10

Impacted Files	Coverage Δ
pybv/io.py	93.05% <100%> (+0.14%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0bc32cc...5ec3c73. Read the comment docs.

[cbrnr]

pybv.write_brainvision now accepts either an (N, 2) or (N, 3) array for the events parameter. The first two columns are always the onsets and the description. The optional third column contains the durations (which are set to 1 sample if this column is not provided, resulting in the previous default behavior).

From my side this is ready to merge. I'll create a PR to extend the functionality of mne.events_from_annotations so that users can easily create the (N, 3) events array. Once this gets merged, I'll update the example in the pybv docs to show how to export events originating from mne.Annotations.

[sappelhoff]

Sounds practical to me, and we don't lose anything by already including this in pybv. Thanks @cbrnr !