-
Notifications
You must be signed in to change notification settings - Fork 525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: BIDS Data Grabber #2174
ENH: BIDS Data Grabber #2174
Conversation
Do we need to duplicate the entire BIDS dataset from pybids? Since pybids is a dependency for this module, you should be able to grab the test data from the original source. I think if you just get |
Oh yeah, good call. Also to-do:
|
Personally, I think it's reasonable to add pybids as a dependency. It's pretty lightweight. |
Right now it is, but we will likely include pandas (and hence numpy etc.) as a dependency of pybids once we start adding transformation and models module. Not sure how people feel about that... |
Codecov Report
@@ Coverage Diff @@
## master #2174 +/- ##
==========================================
+ Coverage 72.26% 72.27% +0.01%
==========================================
Files 1173 1174 +1
Lines 58673 58738 +65
Branches 8442 8454 +12
==========================================
+ Hits 42399 42453 +54
- Misses 14916 14924 +8
- Partials 1358 1361 +3
Continue to review full report at Codecov.
|
@satra comments 9/4/17: Consider adding optional and required outfields, such that an error is thrown is nothing is found for required outfields, and Undefined is passed if nothing found for optional outputs. I'll work on getting travis & circle to work and ping you for a review. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@adelavega I haven't had a chance to make a more thorough review, so here are the couple of comments I've had so far.
nipype/interfaces/bids.py
Outdated
desc='Path to BIDS Directory.', | ||
mandatory=True) | ||
output_query = traits.Dict(key_trait=Str, | ||
value_trait=traits.Dict, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add key and value traits to the internal Dict trait?
nipype/interfaces/bids.py
Outdated
|
||
class BIDSDataGrabber(BaseInterface): | ||
|
||
""" BIDS datagrabber module that wraps around pybids to allow arbitrary |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Docstring should start immediately after class
line, and the contents of the docstring should only be tabbed out 4 spaces, not 8.
"""BIDS datagrabber ...
querying of BIDS datasets.
...
nipype/interfaces/bids.py
Outdated
from warnings import warn | ||
|
||
class BIDSDataGrabberInputSpec(DynamicTraitedSpec): | ||
base_dir = traits.Directory(exists=True, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Directory
is defined in interfaces.traits_extension.py
. Import from interfaces.base
.
nipype/interfaces/bids.py
Outdated
if self.inputs.raise_on_empty: | ||
raise IOError(msg) | ||
else: | ||
warn(msg) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would advocate here for using nipype's logging:
At the top:
from .. import logging
(...)
LOGGER = logging.getLogger('workflows')
And here:
LOGGER.warning('Output key: %s returned no files', key)
nipype/interfaces/bids.py
Outdated
mandatory=True) | ||
output_query = traits.Dict(key_trait=Str, | ||
value_trait=traits.Dict, | ||
desc='Queries for outfield outputs') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should give this field a more sensible default - for example {"func": {"modality": "func"}, "anat": {"modality": "anat"}}. This would save time figuring out the syntax for 80% of users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a fan. The current default of outputting all items is probably almost never usable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here's an other idea for sensible defaults: If no infields are defined, infields for all BIDS entities in project are automatically created, and Undefined
by default. If user connects an input to them, great, if not, they are not passed to pybids. It saves having to define infields.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds like a great idea!
Looks great - thanks for putting this together Alejandro. |
I'm having trouble getting the tests to run because pybids's test data is not getting copied over. In travis (for example), NIPYPE_EXTRAS are installed using -e, like this:
Installing pybids with -e should make the data folder available (which is not installed normally as its not a "package", on purpose). However, it looks like -e is supposed to be used with local files or VCS urls, and when used with pypi it just seems to do a normal install, without coping over the data. Any clues on how to get pip to download the whole package and then install in editable mode, so that non-package folders are available? |
Hi @adelavega, that way you are installing nipype in editable mode, but NOT pybids. To do so, just add some lines after this Line 39 in 4a0bd3d
It will get installed then in editable mode. Please note the closing bracket that originally was at line 39. |
Sweet thanks. I was trying to avoid installing directly from github but if you don't mind, then that works for me! |
Allright, I've incorporated all of your changes and got Travis CI working with the new tests. Any more feedback on "smarter" defaults is appreciated, and any hints on where to inject pybids pip -e installation into Circle. Looks like it's failing because I pip install in the |
Note: I changed the name of the file to |
@adelavega - let's make sure we remove any commit in this tree that adds data to the repo. |
1580c9b
to
97d832b
Compare
Okay @satra I re-wrote history. |
Yay, all tests passing on Circle and Travis. @satra, feel free to make a final review whenever you're ready. |
Sorry @adelavega, yes, that's the right place to insert the installation of pybids 👍 |
|
||
# If infields is None, use all BIDS entities | ||
if infields is None: | ||
bids_config = join(dirname(gb.__file__), 'config', 'bids.json') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as a future improvement to a GrabbitIT interface that could serve as a base class, could this be passed along as a parameter?
this would also allow arbitrary and newer layouts to be used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. The issue here is that I cannot get the entities of a project (or BIDS entities in general), without initializing a BIDSLayout with a specific dataset. So this was the only way I could think of to access the entities definition.
perhaps this should be added to
if you are up for reorganizing otherwise this LGTM |
Great! I've added the reference @satra so feel free to merge. |
Tests will require #2171 to be merged in first. |
@adelavega - i think this is good to go, but just to be sure could you please merge with master? we had to fix a few things. |
@satra done! travis passed so i assume circle will too soon |
Changes proposed in this pull request
infields
outfields
to execute multiple queries, such as grabbing both func and anat files for a subject in a given run.