Adding desc keyword to BIDSPath and get_entities_from_fname?

[adam2392]

Describe the problem

Just starting a convo: it seems that the initial Derivatives PR has been merged into the BIDS specification (https://bids-specification.readthedocs.io/en/stable/05-derivatives/01-introduction.html).

Most explicitly, they seem to add this desc-[label] keyword argument to differentiate different files.

Describe your solution

Add desc keyword to make_bids_basename.

Describe possible alternatives

A separate function called make_derivatives_basename?

Additional context

I'm wondering if there are other ways we can bottleneck how people setup their derivatives folders that adheres to the initial specification? E.g. folder naming, etc.

[jasmainak]

Let's wait a bit. At least until there is a validator PR that is merged, I wouldn't rush on this :)

[sappelhoff]

At least until there is a validator PR that is merged, I wouldn't rush on this

yup, I second that

[adam2392]

I wanted to see if you two might have a different opinion now? Since BIDSPath and get_entities_from_fname have been introduced, it seems like that is the convergent API to handle cutting/slicing a bids related (file)path. However, I have a lot of derivatives that I output and I end up doing hacky code to get it to work with BIDSPath and get_entities_from_fname because I use the desc entity to differentiate the files. E.g. I do this:

def _add_desc_to_bids_fname(bids_fname, description, verbose: bool = True):
    bids_fname, ext = _parse_ext(bids_fname, verbose)

    # split by the datatype
    datatype = bids_fname.split("_")[-1]
    source_bids_fname = bids_fname.split(f"_{datatype}")[0]

    result_fname = source_bids_fname + f"_desc-{description}" + f"_{datatype}" + ext
    return result_fname

You can't use get_entities_from_fname to extract the entities either because it relies on a strict set of entities.
You also can't use BIDSPath because of the same reason. :(

The inclusion of this new entity would be a small addition and simply adds to the entity list we currently have for those two functions. :)

[agramfort]

can you use the check logic to have a more permissive use of the BIDSPath object?

…

[adam2392]

Hi sorry to bring this up again ... :p

I would again like to see if the group thinks this is possible to add in for 0.5. I definitely don't think adding full support of derivatives is possible yet considering the specification is still evolving. The main reason I'm asking again is I have a lot of bug ridden code because I have to add desc- entity into a BIDS path manually by i) separating the basename from the suffix + extension and ii) adding in a description entity and then iii) stitching the suffix and extension back in to adhere to https://bids-specification.readthedocs.io/en/stable/05-derivatives/01-introduction.html#file-naming-conventions

Since BIDSPath is mainly a nice utility pathing object that lets you interchange and update entities very easily via update(description='stuff') function, I wonder if it's possible to just add that into the acceptable list of entities? We could make it explicit that mne-bids doesn't support derivatives IO fully, but BIDSPath has the ability to construct these derivative like filenames?

The change proposed here would allow users to change the naming of the BIDS paths in a more efficient and simple manner.

If acceptable, the PR in #554 is pretty small and lean.

[agramfort]

Isn't the check parameter of BIDSPath the way to go to give your more flexibility?

…

[sappelhoff]

I wonder if it's possible to just add that into the acceptable list of entities?

it's true that it's technically very simple --- but in terms of API it would add an entity that is:

1. not yet in the list of entities ... I have inquired about the reason for that (may simply be an oversight)
2. unclear whether it's only for derivatives or only for raw (to me, but I might just have to read in more detail)
3. generally a bit unclear --> by reading that part about desc-<label>, I get the impression that it is some kind of wildcard in case you can't distinguish two files

point 3. triggers my next question: Why do you absolutely need desc-<label>?

[hoechenberger]

3. generally a bit unclear --> by reading that part about desc-<label>, I get the impression that it is some kind of wildcard in case you can't distinguish two files

Yes, this also caught my attention and worries me a little. I'm afraid supporting this might encourage users to just cram anything into desc without using the "proper" entities, or asking for the standard to be amended. I would also vote for only adding this if we really need it.

[jasmainak]

I get the impression that it is some kind of wildcard in case you can't distinguish two files

I had the same impression but isn't this a problem of the derivatives specification though?

[adam2392]

1. not yet in the list of entities ... I have inquired about the reason for that (may simply be an oversight)

Assuming it is an oversight, my read of the spec is that desc- is a "Derivatives" entity to handle differentiation of derivative transformations from the same raw data. These different derivatives will have the same source entities, so then one definitely needs to distinguish this file path.

2. unclear whether it's only for derivatives or only for raw (to me, but I might just have to read in more detail)

It seems like it's only for derivatives according to https://bids-specification.readthedocs.io/en/stable/05-derivatives/01-introduction.html

[adam2392]

Now that bids-standard/bids-specification#645 is merged in and https://bids-specification.readthedocs.io/en/latest/99-appendices/09-entities.html resolved some of the confusion on this thread with regards to i) is desc an entity and ii) is it for derivatives or raw and iii) what is it used for?

Description is as c/ped from the site:

desc
Full name: Description

Format: desc-<label>

Definition: When necessary to distinguish two files that do not otherwise have a distinguishing entity, the _desc-<label> keyword-value SHOULD be used.

This entity is only applicable to derivative data.

Yes, this also caught my attention and worries me a little. I'm afraid supporting this might encourage users to just cram anything into desc without using the "proper" entities, or asking for the standard to be amended. I would also vote for only adding this if we really need it.

Perhaps some meta checks would help here? For example, two checks can occur:

i) if BIDSPath.description is defined, then what is the extension? If it is a raw manufacturer, raise a warning that description is for derivatives.
ii) if the BIDSPath.root / 'dataset_description.json' file states that this dataset is raw, raise an error.

?

[adam2392]

Is this finally ready to add now that we have write_raw_bids(preload=True) ability :p?

https://bids-specification.readthedocs.io/en/stable/99-appendices/09-entities.html

I would propose:

1. adding in a set of derivative entities to the config
2. add a source_entities and deriv_entities properties to BIDSPath. This would enable parsing file paths like <source_entities>[_space-<space>][_desc-<label>]_<suffix>.<ext> very easily.

[hoechenberger]

2. add a source_entities and deriv_entities properties to BIDSPath. This would enable parsing file paths like <source_entities>[_space-<space>][_desc-<label>]_<suffix>.<ext> very easily.

What do you mean by source_entities? How would you handle "derivatives of derivatives", like "evoked data made from epochs made from band-pass filtered continuous data which was made from Maxwell-filtered raw data"?

[adam2392]

Ah okay derivative of derivatives is a tricky issue. Then maybe none of those properties actually...