Extend method BIDSPath().match() to enable smart path search

[moritz-gerster]

Describe the problem

I love the BIDSPath().match() method for returnigng all bids paths in my root directory as a list. I can then loop over this list to, for example, preprocess all my files.

However, often I would like to filter that list and only loop over some subjects or some tasks. Unfortunately, this is not possible at the moment.

Describe your solution

One could add all the keyword arguments from the BIDSPath class to the match() method. The allowed kwarg types should be both strings (as for BIDSPath) but also list of strings.

If I have subjects=["sub-01", "sub-02", "sub-03"] and I want to loop over the first two, I could get

paths = BIDSPath().match(subject=["sub-01", "sub-02"])

or

paths = BIDSPath().match(task="Rest")

or

paths = BIDSPath().match(task=["Rest", "Move"])

Describe possible alternatives

One could also consider to use the ignore kwargs from mne_bids.get_entity_vals()

paths = BIDSPath().match(ignore_subjects=["sub-03"])

Additional context

No response

[moritz-gerster]

mne-bids/mne_bids/path.py

Line 796 in 1b04da8

`BIDSPath.entities` object. Ignores ``.json`` files.

I also have a use case for not ignoring json files. I would like to save my FOOOF fit data as json files and get the paths using the match method. However, I understand if this will not be supported as BIDS is not defined for FOOOF derivatives.

[agramfort]

just to be sure doing: paths = BIDSPath().match(task="Rest") is the same as paths = BIDSPath().copy().update(task="Rest").match() right? if so you just want to be able to passe a list of task? or subject...

…

Message ID: ***@***.***>

[moritz-gerster]

Yes exactly @agramfort.

If I want to analyze certain files

subjects = ["01", "02", "03"]
tasks = ["Rest", "Move"]
sessions = ["MedOn", "MedOff"]

it would be nice if

paths = []
bids_path = BIDSPath(root=root)
for subject in subjects:
    bids_path.update(subject=subject)
    for session in sessions:
        bids_path.update(session=session)
        for task in tasks:
            paths.extend(bids_path.update(task=task).match())

would reduce to

paths = BIDSPath(root=root).match(task=tasks, subject=subjects, session=sessions)

[agramfort]

I would rather have something like: paths = find_matching_paths(root=root, tasks=tasks, subjects=subjects, sessions=sessions) doing for a parameter singular=plurar does not seem great to me Message ID: ***@***.***>

…

[moritz-gerster]

That would solve the issue 🙂

[agramfort]

great ! could open a PR to do this? thx

…

Message ID: ***@***.***>

[moritz-gerster]

mne-bids/mne_bids/path.py

Lines 817 to 830 in 1b04da8

	# allow searching by datatype
	# all other entities are filtered below
	if self.datatype is not None:
	search_str = f'*/{self.datatype}/*'
	else:
	search_str = '*.*'
	paths = self.root.rglob(search_str)
	# Only keep files (not directories), and omit the JSON sidecars.
	paths = [p for p in paths
	if p.is_file() and p.suffix != '.json']
	fnames = _filter_fnames(paths, suffix=self.suffix,
	extension=self.extension,
	**self.entities)

I don't understand lines 817-822. Why do we search for datatype separately and not with all other self.entities in _filter_fanmes?

[agramfort]

I don't know / remember but this looks like too much magic to me.... What I could suggest is to add parameter to the match method to activate this "exception" or not. def match(self, check=False, *, exclude_json=True, ...) I would also try to comment out these lines and see what test it breaks my 2c

…

Message ID: ***@***.***>

[moritz-gerster]

I now understand this.

_filter_fnames only considers the basenames, not the full path (hence "_filter_fnames" and not "_filter_bidspaths"). datatype is not present in the basename, only in the path, therefore, datatype must be filtered earlier.

bids_path.fpath = 'rawdata/sub-EL002/ses-EcogLfpMedOff02/ieeg/sub-EL002_ses-EcogLfpMedOff02_task-Rest_acq-StimOff_run-1_proc-cleaned_rec-TMSi_channels.tsv'

-> f'*/{self.datatype}/*'

fname = "sub-EL002_ses-EcogLfpMedOff02_task-Rest_acq-StimOff_run-1_proc-cleaned_rec-TMSi_channels.tsv'
-> does not contain datatype=ieeg

[moritz-gerster]

great ! could open a PR to do this? thx

Yes, I'm working on it