write_raw_bids does not take into account changes in raw.info?

[hstojic]

Describe the bug

I load my CTF data as instructed and change some raw.info data but when the data is written these changes are not saved. I'm not sure this is a desired behaviour. I'm using version 0.3.

Steps to reproduce

# load the raw data and edit some info
raw = mne.io.read_raw_ctf(raw_file_name, clean_names = True, preload = False)
raw.info['experimenter'] = pars_gen['info']['experimenter']
raw.info['subject_info']['age'] = pars_ses[session_id]['age']
raw.info['subject_info']['sex'] = pars_ses[session_id]['sex']
raw.info['subject_info']['his_id'] = subject_id

# write
mne_bids.write_raw_bids(
    raw = raw, 
    bids_basename = fname, 
    output_path = dir_megdata_bids, 
    events_data = event_id_array,
    event_id = pars_gen['events']['event_ids'],
    overwrite = True, 
    verbose = True
)

Expected results

If I load again the file I have just saved I would expect raw.info to reflect the changes I have done.

Actual results

Old data, before my changes, is preserved.

Additional information

Platform: Linux-4.11.0-14-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 3.5.2 (default, Nov 12 2018, 13:43:14) [GCC 5.4.0 20160609]
Executable: /home/hstojic/.pyenv/meg3/bin/python3
CPU: x86_64: 16 cores
Memory: Unavailable (requires "psutil" package)
mne: 0.18.2
numpy: 1.16.4 {blas= language = c, lapack= language = c}
scipy: 1.3.0
matplotlib: 3.0.3 {backend=TkAgg}

sklearn: 0.21.2
nibabel: 3.0.1
mayavi: Not found
cupy: Not found
pandas: 0.24.2
dipy: Not found

[jasmainak]

The original files are in fact copied over because we don't have a writer for CTF data. Only the sidecar metadata files are updated. I thought there was a warning somewhere? But if not, we should add one. So, in theory, if you read back the file using read_raw_bids you should get the right information from the metadata files. However, this is not implemented yet AFAIK so you get this behavior. Do you want to give it a shot?

[hstojic]

I see, hence events/annotations are also not made in the data - you get them only with read_raw_bids. Sure, I can give it a try if you point me to a right direction.

On a sidenote, one should not use clean_names = True with read_raw_ctf as channel names will be out of sync in sidecar and data file so read_raw_bids returns an error.

[jasmainak]

So age and sex are written to the participants.tsv file here: https://github.com/mne-tools/mne-bids/blob/master/mne_bids/write.py#L257. I would add it somewhere here: https://github.com/mne-tools/mne-bids/blob/master/mne_bids/read.py#L395. Maybe a private function _handle_participants_tsv ?

On a sidenote, one should not use clean_names = False with read_raw_ctf as channel names will be out of sync in sidecar and data file so read_raw_bids returns an error.

I see, fix for this would also be welcome! It's best if you make an attempt since you are working with the data and can confirm that it works.

[sappelhoff]

I see, fix for this would also be welcome! It's best if you make an attempt since you are working with the data and can confirm that it works.

yes, thanks for reporting this @hstojic ... indeed, the users working with the actual data are best to suggest fixes :-)

[hstojic]

Ok, will give it a try!

On a sidenote, one should not use clean_names = True with read_raw_ctf as channel names will be out of sync in sidecar and data file so read_raw_bids returns an error.

For this I guess there is no fix, just amending the documentation to be careful when using CTF data, right? Where can I amend the docs?

[sappelhoff]

For this I guess there is no fix, just amending the documentation to be careful when using CTF data, right? Where can I amend the docs?

Do we know enough about the situation to say that there can't be a fix? This is the function that gets called when clean_names=True --> https://github.com/mne-tools/mne-python/blob/22dfbb5c55f09b88f595dd5c6d83dac71335df97/mne/utils/misc.py#L194

perhaps it can be fixed (but maybe not) 🤷‍♂️ I would encourage you to investigate. This thread could be 2 PRs:

1. investigating clean_names and either fixing, or putting a warning somethwere
2. enhance read_raw_bids to deal with participants.tsv

[sappelhoff]

closed in #392