[MRG] Conversion of somato data to BIDS

[sappelhoff]

for reference, see this gitter chat

EDIT: check out the conversion script

Current structure:

MNE-somato-data
├── CHANGES
├── code
│   ├── convert_somato_data.py
│   └── README
├── dataset_description.json
├── derivatives
│   ├── freesurfer
│   │   └── subjects
│   │       ├── 01
│   │       │   ├── bem
│   │       │   ├── label
│   │       │   ├── mri
│   │       │   ├── scripts
│   │       │   ├── stats
│   │       │   ├── surf
│   │       │   ├── tmp
│   │       │   ├── touch
│   │       │   └── trash
│   │       └── fsaverage -> /usr/local/freesurfer/subjects/fsaverage
│   └── sub-01
│       └── sub-01_task-somato-fwd.fif
├── participants.json
├── participants.tsv
├── README
└── sub-01
    ├── anat
    │   ├── sub-01_T1w.json
    │   └── sub-01_T1w.nii.gz
    ├── meg
    │   ├── sub-01_coordsystem.json
    │   ├── sub-01_task-somato_channels.tsv
    │   ├── sub-01_task-somato_events.tsv
    │   ├── sub-01_task-somato_meg.fif
    │   └── sub-01_task-somato_meg.json
    └── sub-01_scans.tsv

19 directories, 16 files

[sappelhoff]

It seems like the BIDS naming clashes with the MNE naming convention:

This reminds me of: #4950

[agramfort]

yes this has never been fixed in mne-python but it should not warn for raw bids files

[sappelhoff]

yes this has never been fixed in mne-python but it should not warn for raw bids files

I'll give it a shot

[jasmainak]

can you share link to rendered examples?

[sappelhoff]

can you share link to rendered examples?

sure, but they did not run completely due to the runtime warning that the filename for RAW does not follow MNE conventions:

• https://13544-1301584-gh.circle-artifacts.com/0/dev/auto_tutorials/time-freq/plot_sensors_time_frequency.html#tut-sensors-time-freq
• https://13544-1301584-gh.circle-artifacts.com/0/dev/auto_examples/inverse/plot_dics_source_power.html#ex-inverse-source-power
• https://13544-1301584-gh.circle-artifacts.com/0/dev/auto_examples/time_frequency/plot_time_frequency_global_field_power.html#ex-time-freq-global-field-power

[jasmainak]

ah okay, let us know when that works.

Also I think I prefer to change mne_source_alignment to mne_coregistration

[jasmainak]

I would try to be as much close to BEP003 as possible. So we need to do minimal updates later. For example, you need dataset_description.json:

Derived dataset and pipeline description
In keeping with the BIDS-Raw standard, a dataset_description.json file MUST be found at the top level of the particular pipeline:

Similarly:

Each derivative file SHOULD be described by a JSON file provided as a sidecar or higher up in the hierarchy of the derived dataset

[sappelhoff]

I agree,

the most current BEP for derivatives is here: bids-standard/bids-specification#207

[sappelhoff]

Would these be the correct places to change the naming convention (i.e., accept BIDS endings that are *_meg.fif)

mne-python/mne/io/base.py

Lines 1572 to 1574 in e1a160c

	check_fname(fname, 'raw', ('raw.fif', 'raw_sss.fif', 'raw_tsss.fif',
	'raw.fif.gz', 'raw_sss.fif.gz',
	'raw_tsss.fif.gz'))

and

mne-python/mne/io/fiff/raw.py

Lines 136 to 138 in e1a160c

	check_fname(fname, 'raw', ('raw.fif', 'raw_sss.fif',
	'raw_tsss.fif', 'raw.fif.gz',
	'raw_sss.fif.gz', 'raw_tsss.fif.gz'))

It would also require changing a couple of docstrings and also some docs, such as

• https://github.com/mne-tools/mne-python/blob/master/doc/tutorials/report.rst

[jasmainak]

yep! Don't break the report while you do so. I think it expects certain file format in report.parse_folders() :-)

[codecov]

Codecov Report

Merging #6414 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #6414      +/-   ##
==========================================
- Coverage    89.4%   89.39%   -0.01%     
==========================================
  Files         415      415              
  Lines       74985    74985              
  Branches    12330    12330              
==========================================
- Hits        67039    67035       -4     
- Misses       5118     5120       +2     
- Partials     2828     2830       +2

[sappelhoff]

@larsoner the new somato data is not found for building the docs ... could this be because we are caching it and the cache is still old?

I changed the md5 hash and the URL though:

https://github.com/mne-tools/mne-python/pull/6414/files#diff-00583e36a75a694e071f9fc054a71c08

[larsoner]

The error is:

FileNotFoundError: [Errno 2] No such file or directory: '/home/circleci/mne_data/MNE-somato-data/sub-01/meg/sub-01_task-somato_meg.fif'

When I download your tar.gz and look at the contents I do not see a MNE-somato-data folder but rather a MNE-somato-data-bids folder. Confirmed this is likely the problem with:

$ python -c "import mne; mne.datasets.somato.data_path(force_update=True, verbose=True)"
Downloading archive MNE-somato-data.tar.gz to /home/larsoner/mne_data
...
$ ls ~/mne_data/
...   MNE-somato-data-bids ...

[sappelhoff]

🤦‍♂️ okay, I re-uploaded with proper folder naming. Let's see how this goes.

PS: Thanks a lot for spotting it

[jasmainak]

@sappelhoff did you run the dataset through the validator? any issues you faced? Is it compliant?

otherwise PR looks clean to me.

[sappelhoff]

yep, see:

(base) stefanappelhoff@arc-lin-004309:~/Desktop$ bids-validator MNE-somato-data
This dataset appears to be BIDS compatible.

        Summary:                  Available Tasks:        Available Modalities: 
        11 Files, 334.48MB        somato                  T1w                   
        1 - Subject                                       coordsystem           
        1 - Session                                       channels              
                                                          events                
                                                          meg                   


	If you have any questions, please post on https://neurostars.org/tags/bids.

I also added dataset_description.json files for the derivatives ... they are not completely compliant (e.g., the freesurfer directory is still as output by recon-all), but as much as is reasonable given the WIP state of derivatives.

[jasmainak]

sounds good. +1 to merge from my end once CIs are happy

[teonbrooks]

just curious @sappelhoff, are you adding these derivative files to the BEP 021?

[sappelhoff]

@teonbrooks not yet - but we just had a BIDS discussion at OHBM and we'll get more focused on the derivatives starting soon :-)

... probably with a survey of what most people need (doing a "data driven 80-20" as @jasmainak put it ;-) )

[sappelhoff]

@agramfort with using anchors: do you mean I should do it like this? (see below)

.. _`somatosensory dataset`: https://martinos.org/mne/stable/manual/datasets_index.html?#somatosensory

=========================================
Compute source power using DICS beamfomer
=========================================

Compute a Dynamic Imaging of Coherent Sources (DICS) [1]_ filter from
single-trial activity to estimate source power across a frequency band. This
example demonstrates how to source localize the event-related synchronization
(ERS) of beta band activity in the `somatosensory dataset`_

[agramfort]

you need to insert .. _somato-dataset: in L124 in datasets_index.rst and then you'll be able to refer to this section with :ref:`somato-dataset`

[jasmainak]

I'm blocking the PR so as to remind us that we should use the digitization coordinates as in the spec instead of the trans files.

[sappelhoff]

you need to insert .. _somato-dataset: in L124 in datasets_index.rst and then you'll be able to refer to this section with :ref:somato-dataset
--

to remind myself, this is the rst docs section dealing with that: https://www.sphinx-doc.org/en/1.5/markup/inline.html#cross-referencing-arbitrary-locations

[sappelhoff]

rendered:

1. https://14457-1301584-gh.circle-artifacts.com/0/dev/auto_examples/inverse/plot_dics_source_power.html
2. https://14457-1301584-gh.circle-artifacts.com/0/dev/auto_examples/time_frequency/plot_time_frequency_global_field_power.html
3. https://14457-1301584-gh.circle-artifacts.com/0/dev/auto_tutorials/time-freq/plot_sensors_time_frequency.html

[agramfort]

besides subj -> subject LGTM

[jasmainak]

Thanks @sappelhoff !

[massich]

I know I'm late on the review cycle and this is merged, but do you guys find worth to create a helper function that allows composing the name?

def fname_helper(subject, task='somato', data_type_prefix=''):
    return op.join(
        'sub-{0:02d}'.format(subject),
        data_type_prefix,
        'sub-{0:02d}_task-{}_meg.fif'.format(subject, task)
    )

from mne.datasets.somato import data_path, fname_helper

raw_fname = op.join(data_path(), fname_helper(subject=1, data_type_prefix='meg'))
fname_fwd = op.join(data_path, 'derivatives', fname_helper(subject=1))

[massich]

ups !! the file names are not the same.

one is _meg.fif the other one is _fwd.fif

[jasmainak]

The derivatives is not set in stone, so I would not make a new function. There is already one in mne-bids for the existing BIDS spec: https://mne.tools/mne-bids/generated/mne_bids.make_bids_basename.html#mne_bids.make_bids_basename.

[massich]

Also reading the artifacts I found that we are using a private function in a tutorial:

mne-python/examples/time_frequency/plot_time_frequency_global_field_power.py

Line 56 in cc836c2

from mne.stats import _bootstrap_ci

Shall we make the function public? wrap wherever is exposed and make this one public?

[agramfort]

Also reading the artifacts I found that we are using a private function in a tutorial: https://github.com/mne-tools/mne-python/blob/cc836c2635b7a720ec6c60074a1997fe52d7ca2f/examples/time_frequency/plot_time_frequency_global_field_power.py#L56 Shall we make the function public? wrap wherever is exposed and make this one public?

nice catch ! this is evil. @sappelhoff can you fix? or anyone?

[sappelhoff]

Can @massich elaborate a bit on his intention for the fix?

[jasmainak]

you're not supposed to use private functions in examples ...

[sappelhoff]

Yes, I learned that repeatedly through you and Alex :-D

But I was wondering what @massich meant with

wrap wherever is exposed and make this one public?

[jasmainak]

well if you wrap another function around it, you can potentially expose fewer parameters publicly while still keeping the flexibility of the private function. Not sure if that's necessary though.

[larsoner]

@sappelhoff looks like the -5120-bem.fif and -5120-bem-sol.fif files are missing from the bem directory. Was this intentional or just an oversight? If the latter, I could add them but do I need to do anything at the BIDS end? FWIW this can be used to create them:

# in ~/mne_data/MNE-somato-data/derivatives/freesurfer/subjects
model = mne.make_bem_model('01', conductivity=(0.3,), subjects_dir='.', verbose=True)
mne.write_bem_surfaces('01/bem/01-5120-bem.fif', model)
bem = mne.make_bem_solution(model, verbose=True)
mne.write_bem_solution('01/bem/01-5120-bem-sol.fif', bem)

[britta-wstnr]

Can confirm -- tried to run some older code after updating somato. The BEM solution seems to be missing.

[sappelhoff]

Was this intentional or just an oversight?

sounds like an oversight to me because I have no clue what these files do.

This is the conversion script: https://github.com/mne-tools/mne-scripts/blob/master/bids_conversion/convert_somato_data.py

As long as the files are being put into the derivatives directory, it does not hurt BIDS. But the best would probably be to fix the conversion script.

[larsoner]

Okay, will push a commit there and update the dataset on OSF

[sappelhoff]

Okay, will push a commit there and update the dataset on OSF

thanks a lot @larsoner and sorry to cause problems.