[ENH] BEP044 - Stim-BIDS

[neuromechanist]

This PR addresses #153 by introducing specifications for handling and standardizing stimulus files and their annotations within the BIDS specifications. The changes focus on improving the organization, referencing, and metadata of stimulus files to enhance consistency, reusability, and efficiency.

Important

HTML render of the stimuli page : https://bids-specification--2022.org.readthedocs.build/en/2022/modality-agnostic-files/stimuli.html

Note

The issue: #153
Google Doc
Example PR: bids-standard/bids-examples#433

Known issues:

• The validator fails because extensions in /src/schema/rules/files/raw require datatype. Stimuli might be a special data type that can only be present at the root of the dataset. So, the datatype field is missing for now.
• There are some style errors by the remark validators

cc: @bids-standard/bep044 and @monique2208

[Remi-Gau]

will do a bit of clean up to get less red in CI and maybe see if we can get the HTML version of the BEP to render

[Remi-Gau]

HTML: stimuli page
https://bids-specification--2022.org.readthedocs.build/en/2022/modality-specific-files/stimuli.html

[yarikoptic]

initial quick pass with little recommendations etc

[yarikoptic]

@neuromechanist can you please update the PR merging the #2367 and also addressing the conflicts . What is your plan towards making it available for review?

too old

[neuromechanist]

Quick update on the recent pushes, now that @yarikoptic's media-files harmonization is merged here:

• The schema caught up with the spec text: the stim/annot entities are properly wired in, the stim_id checks are rewritten so the validator can actually run them (value patterns now live in the column definitions), and stimulus formats are consistent across the appendix, the file rules, and the docs.
• /stimuli now supports hierarchical subdirectories. A subdirectory is self-describing, so a stimulus set can move between datasets as a portable unit. A dataset-wide stimuli.tsv at the root is RECOMMENDED, even a sparse one listing only the stimulus_ids; scoped catalogs amend or override it following the inheritance principle.
• Legacy datasets are untouched: the new rules only kick in when a stimuli.tsv exists somewhere under /stimuli.
• Everything is tested end to end with a new beh_stimuli example ([WIP] examples for BEP044, Stim-BIDS bids-examples#433, now including LLM-generated descriptions and HED annotations for the NSD shared1000 images) and a validator implementation (feat: Support BEP044 stimulus organization in /stimuli bids-validator#445); legacy examples like eeg_matchingpennies validate unchanged.

[yarikoptic]

initial, mostly formatting, but raising concern to added groupping folders which are nohow formalized or mostly not inline with the main bids hierarchy building principles (besides hardcoded datatype/ folders)

note that so far BEP does not introduce the concept of stimuli NEAR the data files as we need for reprostim

[yarikoptic]

oh, this is "new" to me concept here, somewhat mimicing the "datatype/" folders we have now (anat/, func/ etc) which sticks out like a sore point... yet to digest this and see how to mediate -- since unlike datatype those are nohow formalized. I feel like we would need some 'cohort-' like entity (similar to ses-) to group them and providing cohorts.tsv to describe such groups if we are to go this way

[yarikoptic]

note that here must be following Inheritance principle but we have not defined it well for .tsv yet!

[yarikoptic]

There are tsv types

src/common-principles.md:```tsv {linenums="1"}
src/common-principles.md-onset      duration        response_time   trial_type      trial_extra
src/common-principles.md-200        20.0    15.8    word    中国人
src/common-principles.md-240        5.0     17.34e-1        visual  n/a
--
src/common-principles.md:```tsvgz {linenums="1"}
src/common-principles.md-200        20.0    15.8    word    中国人
src/common-principles.md-240        5.0     17.34e-1        visual  n/a

hence

Suggested change

	```Text
	```tsv

[yarikoptic]

likely this is intended to be yet another tsv inlined example, not .md table right away, reformat

[yarikoptic]

also tsv

[yarikoptic]

also reformat as tsv inline

[neuromechanist]

Thanks @yarikoptic,

Deferring the styling convo to later:
I think we briefly discussed having subdirectories in the stimuli/ dir. Having thousands of images or their annotations, could make subdirs quite useful. I still proposed it as a convenience though, meaning that validators and tools flatten them.

On the TSV inheritance I agree that it is loosely defined, although I doubt it would be hard to tighten with examples?

[yarikoptic]

Deferring the styling convo to later: I think we briefly discussed having subdirectories in the stimuli/ dir. Having thousands of images or their annotations, could make subdirs quite useful. I still proposed it as a convenience though, meaning that validators and tools flatten them.

I do feel that directories could be useful but I just noted that it adds some "inconsistency". We should also check on what they do in other related BEPs like BEP036 on that

On the TSV inheritance I agree that it is loosely defined, although I doubt it would be hard to tighten with examples?

I would say we first need to tighten in language since examples can't mandate how things are validated etc, they could just demonstrate. And overall point here, as .tsv as .json just different containers for metadata, it should be congruent with IP overall as there is now no "overrides" but rather only "summarization" into higher levels... the question is either it is on per-row or per-cell basis -- we better make it clear first in common principles outside of this BEP/examples, and here only demonstrate... what example use cases you had in mind?