[WIP] allow read_raw_bids to take additional input parameters #244
Conversation
sappelhoff
changed the title
Codecov Report
@@ Coverage Diff @@
## master #244 +/- ##
==========================================
+ Coverage 98.41% 98.43% +0.02%
==========================================
Files 15 15
Lines 1573 1597 +24
==========================================
+ Hits 1548 1572 +24
Misses 25 25
Continue to review full report at Codecov.
|
|
I don't quite understand this PR. Shouldn't BIDS structure tell you where these additional files are? You shouldn't have to expose additional arguments for this. |
| if ext in ['.con', '.sqd']: | ||
| read_params = dict(elp=opt.electrode, hsp=opt.hsp, hpi=opt.hpi) | ||
| elif ext == '.pdf': | ||
| read_params = dict(config_fname=opt.config, head_shape_fname=opt.hsp) |
There was a problem hiding this comment.
you shouldn't have to specify them. It should come from BIDS
There was a problem hiding this comment.
this gets passed to _read_raw so it's not yet in BIDS AFAIK.
There was a problem hiding this comment.
This is the write utility of the CLI ... so the data is not yet in BIDS and has to be supplied "manually"
| parameters of MNE-Python's read_raw_* function for the data format you | ||
| intend to read. For example, when reading fif data you can specify | ||
| `read_params` as {allow_maxshield=False}. Non-fitting parameters will | ||
| be ignored. Not all parameters are currently supported. If None, |
There was a problem hiding this comment.
+100. Don't silently ignore things.
There was a problem hiding this comment.
I am raising Userwarning in verbose mode ... should I
- always raise a warning?
- always throw an error?
- ... do something else? what?
| The keys of the dictionary must correspond with potential input | ||
| parameters of MNE-Python's read_raw_* function for the data format you | ||
| intend to read. For example, when reading fif data you can specify | ||
| `read_params` as {allow_maxshield=False}. Non-fitting parameters will |
There was a problem hiding this comment.
I wouldn't bother adding a new argument just for this. Let's do use-case driven software engineering. Ask people around you to try mne-bids and raise issues. It will give you a sense of gratification but also give a sense of priority.
There was a problem hiding this comment.
fully agreed. Also besides allow_maxshield all the other params contain info that should be present in the BIDS sidecar files.
There was a problem hiding this comment.
see my response: #244 (comment)
| if ext in ['.con', '.sqd']: | ||
| read_params = dict(elp=opt.electrode, hsp=opt.hsp, hpi=opt.hpi) | ||
| elif ext == '.pdf': | ||
| read_params = dict(config_fname=opt.config, head_shape_fname=opt.hsp) |
There was a problem hiding this comment.
this gets passed to _read_raw so it's not yet in BIDS AFAIK.
| The keys of the dictionary must correspond with potential input | ||
| parameters of MNE-Python's read_raw_* function for the data format you | ||
| intend to read. For example, when reading fif data you can specify | ||
| `read_params` as {allow_maxshield=False}. Non-fitting parameters will |
There was a problem hiding this comment.
fully agreed. Also besides allow_maxshield all the other params contain info that should be present in the BIDS sidecar files.
| parameters of MNE-Python's read_raw_* function for the data format you | ||
| intend to read. For example, when reading fif data you can specify | ||
| `read_params` as {allow_maxshield=False}. Non-fitting parameters will | ||
| be ignored. Not all parameters are currently supported. If None, |
There was a problem hiding this comment.
+100. Don't silently ignore things.
I agree, we should support that in the future. Let me talk only about MEG first: It will be a major effort: We have to rely on these BIDS files:
and from these files, we have to extract the data that we need for a given data format:
(what about so we need to write reading functions for the BIDS files ... and then code to ...
I would not take this on currently, but rather wait for a feature request or user issue. This present PR however provides an intermediate solution, until the "real deal" is implemented. |
|
One last comment: I would separate this discussion between MEG and EEG, because the situation seems much simpler in EEG. in EEG, we have The Reading these would be quite simple, and if we have LPA, RPA, Nasion, and the electrode positions with respect to these three landmarks, we can easily convert this into a Neuromag coordsystem as used in FIF, and we'll get rid of all problems :-) |
|
FYI there is current a rewamping of the digitization / montage mechanism by @massich the hsp, elp etc... should be handled outside of the read raw function. Vision is to be able to read them outside and then call a set_montage method on the raw. |
yes, I really like that - do we know something about the timescale when this will be finished? |
|
Hopefully in a few weeks
And before the next release
If you have some working code or not using eeg and montage with mne I am sure @massich would love to know
|
|
@sappelhoff I would simplify this code. See something like this: by doing this no need to check for ignored params. It's easy to document as you need to point to mne-python documentation etc. |
yes, but if not a GSoC, when will we undertake such a major effort? :) Then again, I think you should focus your efforts on the study template so you are the user and you figure out what are the issues. We have so far not hit this issue in the study template, or have we?
agreed.
No, your headshape.pos should do the job
These follow the BIDS naming specifications, e.g., sub-control01_ses-001_task-rest_run-01_markers.<mrk,sqd> as in the specification example.
you'd simply read in a text file, so that should not be too difficult? Based on the link you posted, it seems to me that you just need to write a reader for CTF. We can ignore KRISS for now and for the rest you already have the digitizer info in the files themselves.
hmm ... maybe, but I'm not too enthusiastic about this solution. |
fair enough :-)
ok
we hit it once, when we wanted to plot sensors, but Alex then fixed it by changing an argument and it's fine now.
okay, let's be pragmatic: Should I invest some more work and incorporate alex's feedback and then we merge, or should we close this? |
|
drive by comment:
is there a way to set allow_maxshield with FIF objects without adding a param? |
|
Thanks for weighing in @teonbrooks :-)
are there any cosmetic things though? The main refactoring of this PR is the introduction of the However, if we are not exposing Please advise:
|
|
Again I would just add a read_params dict() param to read_raw_bids saying
that these are kwargs
that can be passed to mne read functions cf mne documentation
… |
@sappelhoff this is my top priority right now. And the sooner I get done with it the better. Ping me anytime. |
that makes sense. this is me playing the 4 year long game of mne-tools/mne-python#1930 😄 |
|
Let me take over this PR for a moment, so that @sappelhoff you can concentrate on the study-template :) You can review this when you are bored. |
| mrk=hpi, preload=False) | ||
| mrk = _find_matching_sidecar(bids_fname, bids_root, | ||
| '_markers.???') | ||
| raw = io.read_raw_kit(raw_fpath, elp=None, hsp=None, |
There was a problem hiding this comment.
mne-bids only copies the mrk file but mne-python expects also elp and hsp to be provided. This tells me that the writing of this file is already broken. cc @sappelhoff
jasmainak
changed the title
|
@agramfort I'm fine with adding **kwargs if you insist ... so @teonbrooks finally has what he wished for 4 years ago ;-) But, how do you propose to deal with conflicting kwargs? E.g.: Also, I feel doing this will hide some of the issues that you have in BIDS and mne-bids. For example, when I wrote the tests for reading the @sappelhoff I left some todos for you. I think the PR is back to you as I won't have more time :-) Sorry I made a bit of mess while I was playing around, feel free to add issues and revert my commits if that's more efficient. |
|
@agramfort <https://github.com/agramfort> I'm fine with adding **kwargs
if you insist ... so @teonbrooks <https://github.com/teonbrooks> finally
has what he wished for 4 years ago ;-)
But, how do you propose to deal with conflicting kwargs? E.g.:
read_raw_edf has an argument called eog and then we also read the channel
types from the BIDS sidecar files. If you don't expose this parameter, it's
clear that the BIDS sidecar should be used to set the EOG channel. However,
now there is a conflict.
use the **kwargs at your own risks and the BIDS files should always have
the priority.
Also, I feel doing this will hide some of the issues that you have in BIDS
and mne-bids. For example, when I wrote the tests for reading the hsp
files in the KIT reader, it's when I realized that all the files necessary
are not written by mne-bids.
you need to use the software your write to understand how / when it breaks
:)
|
| ('y', y), | ||
| ('z', z)]) | ||
| _write_tsv(fname, data, overwrite=overwrite, verbose=verbose) | ||
| return fname |
There was a problem hiding this comment.
I think this is slightly problematic, because like this, we will write electrodes.tsv data that might originate from an idealized Montage as opposed to a measured DigMontage 🤔
for BIDS data, we only want to write electrode positions (x,y,z) if these have been measured/digitized ... not some template or idealized positions
cc @jasmainak
I'll have to figure out if it's possible in MNE to distinguish between a DigMontage and a Montage ... currently it seems like once they are set, they are treated equally.
Ideally, I would vote for only writing electrodes.tsv files if we know that
- the electrode positions are MEASURED
- we have NAS, PRA, LPA to scale the positions to FIFF space
There was a problem hiding this comment.
why is that? Does the BIDS specification make this mandate somewhere? I couldn't find it with a quick search ...
There was a problem hiding this comment.
i don't think much consideration has been given toward template data in the digitization in the bids discussion. i think it would be nice to have this information but if the analysis ultimately rest on the template, the digpoint data should exist somewhere (in the electrode file or in the attached template). it would be incumbent on the researcher to describe it in their methods somehow.
There was a problem hiding this comment.
To me either you have it described in your electrode file, if you have a digitization or in the template if you are using a template. (and the template structure should be a replica of electrode file).
There was a problem hiding this comment.
I feel like researchers / data-providers should only provide measured data (=digitized, true sensor positions) ... template positions etc. should be provided by the analysis software.
Sharing template positions without an indicator that the positions in question are in fact just some "standard", "idealized" positions can easily lead to misunderstandings, where somebody thinks that these are true positions (...and this would lead to errors in source localization)
With this reasoning, a researcher would write an electrodes.tsv and accompanying coordsystem.json only if they actually have true positions ... else, they just omit this info and the analysis software can provide a template based on the data format.
Think about it this way: If researchers / data-providers also shared an electrodes.tsv for template positions for each dataset and datafile, we'd have a lot of duplication and clutter ... because the template positions are always the same, depending on the sensor name and manufacturer.
There was a problem hiding this comment.
I assume this is true for all kinds of sensors (templates can almost always be unambiguously guessed)
This is not true because if this were true, you would not need raw.set_montage method or montage argument in the raw readers. As we are practical people, I invite you to try writing code to unambiguously guess it ;-) If it cannot be guessed in 5 tries, then the burden of sharing the electrode locations should lie on the researcher sharing the data, no?
There was a problem hiding this comment.
If it cannot be guessed in 5 tries, then the burden of sharing the electrode locations should lie on the researcher sharing the data, no?
okay, fair enough --> I already agreed that this could/should be fixed in BIDS (by adding a key-vaul pair whether the electrode locations are template positions or digitized positions)
but not everybody is using BIDS, so we will also have to deal with this in MNE-Python. The suggestion by @larsoner seems good to me at a first glance (using BOTH his points)
There was a problem hiding this comment.
just fyi, this is what Robert Oostenveld thinks about this: bids-standard/pyedf#7 (comment)
... the thread might be an interesting read for everybody working on digitization in MNE-Python
There was a problem hiding this comment.
For me BIDS (MEG, EEG and iEEG) so far have always been on raw (i.e. measured and unprocessed) data, not on processed (template) or model data (like electrodes on a sphere). That might have had to be stressed more.
I would argue that this should be fixed in the BIDS specification not in MNE. If you share template positions, there should be clear indicator in the specification that it's a template. Ultimately, many researchers won't care as long as they can run their pipeline.
Agreed, I propose that the BIDS spec for raw data is updated to clarify that people should include measured data, not templates. Should that also be extended to template anatomical MRIs? I.e. would people right now also be including for example colin27 as anat/sub-01_T1w.nii, in case they don't have an anatomical MRI?
If you share the electrode locations only in some BIDS datasets, how is the researcher supposed to guess which template it belongs to?
That has been given explicit thought and is part of the specification: you SHOULD specify CapManufacturersModelName and EEGPlacementScheme. These are more precise than merely relying on electrode names, since cap designs vary between manufacturers and sometimes between Caucasian/Asian fit. If you (as experimenter) tell me what cap you used - I (as software developer) can provide you with the template - which I can also update and improve over the years. E.g. I might start with a spherical 10-20 template, then in a later version of the software update that for a more accurate one. Ideally EEG cap manufacturers would get themselves a digitizer, scan all of their cap designs and share those (as we are arguing for here).
There was a problem hiding this comment.
I opened an issue for this on BIDS-Specification: bids-standard/bids-specification#318
for MNE-BIDS, we "solved" this in our last call, I summarize the solution here: #264
|
closing, because part of the work will be done in #263 and the rest of this PR can be trashed because it's not (yet) reliant on BIDS. Feel free to re-open if somebody plans to work on this in the near future - but I won't have the time for this particular thing |
closes #203
Merge checklist
Maintainer, please confirm the following before merging: