-
Notifications
You must be signed in to change notification settings - Fork 154
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Standardize .bidsignore #131
Comments
Seems sensible... like the use of .gitignore syntax and not unifying with .gitignore itself. |
+1 though as long as we describe our ignore syntax explicitly it is IMO not strictly necessary to have it as a subset of |
You write
I have a conceptual issue there... but perhaps I am just stating the obvious. with For me Can we agree on Also note that if I were to release my application/pipeline as "myapp", and if it would need to ignore files, I might want those be listed in a |
My assumption would be that .bidsignore would be used by the bids-validator
and pybids (the python package that many python-based bids-apps are using
for parsing the BIDS folder structure). Are there any others?
…On Wed, Jan 23, 2019 at 4:52 AM Robert Oostenveld ***@***.***> wrote:
You write
support of .bidsignore file across software toolkits
I have a conceptual issue there... but perhaps I am just stating the
obvious.
with .gitignore I know that it is git (the application) that is reading
it and subsequently ignoring certain files. With .bidsignore it is not
explicit which application should ignore the files. I expect bids-validator
to ignore it (and know that expectation to be met). But if have a
.bidsignore in my own BIDS dataset, I don't expect my own pipelines to be
ignoring the listed files. Most likely I put the to-be-ignored datafiles
there just *because* my application/pipeline needs them.
For me .bidsignore conceptually corresponds to the "diff" between the
file list that the one application (bids-validator) expects, and the file
list that the other application (my pipeline) expects. That works in the
specific case, but fails to scale if *three* applications are involved. I
also don't see how it would scale to arbitrary applications.
Can we agree on .bidsignore only to be read and interpreted by the
bids-validator? Or more precisely, that it is the list of files (and
directories) to ignore to make the dataset bids compliant. Compliance
testing is currently implemented in the nodejs bids-validator app, but in
the future might be checked by other toolkits.
Also note that if I were to release my application/pipeline as "myapp",
and if it would need to ignore files, I might want those be listed in a
.myappignore. As such it would fall outside the bids-specification (since
future apps cannot be part of the current specification). The solution to
keep the dataset bids-compliant would then be to include .myappignore in
the .bidsignore.
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<#131 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AMl59qyQkdVQDLa-rw9ROFQ14_xgt6C1ks5vGDDbgaJpZM4aNYRV>
.
|
I agree with @robertoostenveld . However, regarding this:
Do you mean that these toolkits are completely equivalent implementations? That is, for the same input they produce the same validation output. Otherwise, I can imagine a scenario where one dataset would pass on one validator but not on the other. Thus, I think there can be only one validator ... |
I would say - "No", because it is is not a
|
wouldn't you say that this requires a fix in the validator rather than bending the standard to make the validator happy? Perhaps a command line flag like Then you can have |
This is starting to get complicated, IMO, which worries me. I think With bids-validator, that's easy: Its whole purpose in life is to throw errors. With pybids, it's a little less straightforward, as I may mean "Do not index" or "Index, but don't yell about non-standard entities". I think we have strictness modes in But I really don't like the idea of a That said, there's nothing stopping you from adding |
@yarikoptic - but and to reduce output noise from the validator just as .git and others do to reduce noise on their output. so it's tied to the spec and hence to validation of the dataset. |
Was it? |
@effigies et al. I agree that we should stop talking about separate application specifics here and concentrate on the .bidsignore to get described in the spec as to prescribe files/directories which must not be considered as far as we are talking about BIDS (even if conform bids, possibly in a subdirectory structure) - queries, manipulations, etc. So mentally really the same what .gitignore does for git. |
Just to return to this, I want to copy in the And my overall position is I would rather have explicit rules than a dependency on git, which could potentially (though is unlikely to) be a moving target.
I copied that description here:
|
I thought I commented on this earlier, but I guess not. Just to add my two cents: I feel pretty strongly that |
@tyarkoni Could you please elaborate a bit more since I am not sure I am following.
Are you suggesting that PyBIDS could/should use bids-validator (the alien nodejs utility) to provide it (PyBIDS) with the list of ignored (or the opposite - legit) files, so that PyBIDS could then use that information to filter its results?
it validates consistency of lengths for the same func/ file across subjects/runs - so clearly not just independently validates files in isolation.
E.g. if I have a file (or a subtree of directories, e.g.
How would that happen? validator would just ignore them, while Python packages would have no clue they were ignored (unless the idea for them to talk to validator)? |
Each
Well, the JavaScript validator may be doing this... I don't believe the Python validator is (which is a problem that probably needs to be addressed separately).
The
On this proposal, the |
On re-read, @yarikoptic, I think what you may be missing is that the Python validator doesn't work like the JS validator. It doesn't do a single pass through the dataset and return problems; it currently validates each file on an ad-hoc basis. We definitely need to implement layout-level checking too (and that should probably live in the |
ATM bids-validator supports
.bidsignore
file and describes it as.bidsignore
Optionally one can include a
.bidsignore
file in the root of the dataset. This file lists patterns (compatiblewith the .gitignore syntax) defining files that should be ignored by the
validator. This option is useful when the validated dataset includes file types not yet supported by BIDS specification.
while also hardcoding some additional ignores:
In the Python land of pybids there is a constant battle to hardcode various additional ignores, see e.g. bids-standard/pybids#277 (comment) and a common consensus is that supporting
.bidsignore
would be the way to go forward. For that to happen properly, and to not have varying/possibly conflicting support of.bidsignore
file across software toolkits, it is desired to describe that in BIDS specification document somewhere, e.g. a new99-appendices/09-software-support.html
, stating our expectations for that file syntax. I see two choices.gitsignore
- sounds like a nice idea, but I am afraid it might be also a fragile one unless we start use git itself, i.e. git-check-ignore (with hard-dependency on git, which I am not) since through time Git might introduce/change .gitignore specification. ATM (if I got it right) bids-validator relies on https://www.npmjs.com/package/ignore to provide JS implementation. For pybids we could probably find some Python implementation, but so far the ones I found are too adhoc and/or old/abandoned (e.g. https://github.com/snark/ignorance), so calling out togit-check-ignore
sounds like a best idea.gitignore
- we clearly describe what patterns are supported etc.What do you think the @bids-standard/everyone ?
The text was updated successfully, but these errors were encountered: