Filtering, selecting, and renaming data in _ChRIS_ plugins

[jennydaman]

Often there is a need to filter data found in the input directory. Examples:

1. I want to select all .nii.gz files (and omit unmatched files such as .json files)
2. I have a DICOM study consisting of several DICOM series. I want to process all T2w HASTE
   MRI scans. Concretely, I need to match the keywords "T2_HASTE_4 CONCAT LONG TR" in the ProtocolName DICOM tag.
3. I want to select some .nii.gz files depending on what DICOM tags are specified in their BIDS sidecar¹².
4. I want to select all files greater than 1MB in size.
5. I want to rename files.

Use case (1) can be achieved using pl-topologicalcopy, and the storage use has been optimized in #545. However it is a clumsy solution:

• --filter is specified as a regex
  • , (comma) is a special character, cannot filter on commas in filenames. Related to Add protection against files/folders with paths that contain commas #609
• (Subjective) the terminology "topology synthesis" is confusing. Unnecessary jargon harms our project.

Use case (2) can somewhat be solved. There is a convention on the clinical side that SeriesDescription contains ProtocolName and in ChRIS, folders containing DICOM series are named after SeriesDescription. However, relying on conventions is brittle, so use case (3) is a more correct approach.

Use cases (3), (4), and (5) can be solved by writing a (ds-type) ChRIS plugin, e.g. pl-bulk-rename. However. this causes files to be copied, consuming twice the amount of storage than necessary.

The problem and its context can be summarized like this: simple operations (filter, select, rename) lead to wasteful storage consumption. Some of these use cases are solved by pl-topologicalcopy, but its usage is esoteric and limited.

Solution A: expand the capabilities of ts-type plugins to cover the use cases of filter, select, and rename. Pros: efficient solution which is continuous with current design. Cons: it's often easier for end users to write a custom ChRIS plugin than to conceptualize and apply ts-type plugins for the same purpose.

Solution B: #515. Pros: plugin-centric design emphasizes that ChRIS plugins do everything, simpler to understand and audit ChRIS pipelines. Cons: computationally expensive.

Solution C: provide a way for plugin developers to "annotate" or provide "hints" to CUBE that an output file is a copy of an input file. Pros: plugin-centric design and efficient. Cons: optimizations are unlikely to be used by novice plugin developers.

Footnotes

1. https://bids-specification.readthedocs.io/en/v1.4.0/05-derivatives/02-common-data-types.html ↩

2. https://brainvoyager.com/bv/doc/UsersGuide/GettingStarted/NIfTIAndJSONSidecarFiles.html ↩

[rudolphpienaar]

Let me rephrase this idea in a slightly different way to check whether I’m understanding it correctly.

It seems like the missing CUBE abstraction is a storage-efficient “file view” or “slice” of an existing output tree.

Today, if I want a subset, rename, or regrouping of files, the practical path is often to run a plugin that materializes a new output directory. That works, but it turns cheap logical operations into duplicated storage.

Would one concrete version of Solution C be something like this?

1. A plugin produces a small file-view manifest.
2. The manifest maps existing source files to desired output paths.
3. CUBE consumes that manifest during output registration.
4. CUBE validates the mapping.
5. CUBE satisfies the view by creating ChrisLinkFile artifacts in the plugin output directory.
6. Users see a normal-looking output tree.
7. Downstream plugins still work because pfcon already understands .chrislink files and materializes them into the plugin input directory.

Example manifest:

views:
  - source: some/previous/output/a.nii.gz
    output: selected/a.nii.gz
  - source: some/previous/output/b.nii.gz
    output: selected/b.nii.gz

That seems like it could support two related use cases with one common CUBE mechanism:

• A better ts-style file-view plugin, perhaps pl-topofileview, in the same general space as pl-topologicalcopy, but with clearer UX around selecting, filtering, renaming, and regrouping outputs from existing plugin instances.
• More general plugins that inspect real input data and emit the same manifest.

So maybe the important thing is not whether the producer is ts or ds, but that CUBE has one common file-view registration mechanism.

This also seems to fit an interactive/plugin-assisted UX. For example, the UI could let a user browse a parent plugin’s output tree, select files/folders, define filters or rename rules, preview the resulting view, and then click “commit.” The commit action would still create a normal child plugin instance, e.g. pl-topofileview, with the UI-generated file-view spec as input/ parameters. So the UI is just a spec builder; the committed result remains a normal plugin instance in the feed graph, preserving provenance and making the operation reproducible.

This would be a QoL improvement and a storage-efficiency improvement. It also seems aligned with the seagap model, and potentially useful for ATLAS workflows where curated views/slices of larger output trees are common.

[jennydaman]

Your proposed concrete Solution C seems reasonable. Here's another idea: ChRIS plugins can create symlinks from inputdir to outputdir. When you run this plugin on-the-metal, it works as it does. When you run this plugin via CUBE, when CUBE is registering output files, it detects that the output files are symlinks which point to the inputdir, and translates those symlinks to .chrislink resources.

My alternative idea embodies the idea that ChRIS plugins are "just LInux programs" which work outside of ChRIS, is interoperable, and easy for novice plugin developers to implement without needing domain-specific knowledge about ChRIS behavior nor read the docs about the manifest's schema.

[jennydaman]

Also yes, a UI-generated file-view is something we need. A code-based solution which can be used in pipelines is also needed.

This discussion is motivated by my current work on the fetal-recon pipeline. Related: FNNDSC/Fetal_Brain_MRI_Reconstruction_Pipeline#1

[rudolphpienaar]

Ooohhh... I like that -- use OS symlinks at the plugin level and, if run in ChRIS, CUBE converts these to .chrislinks. I like it. I'm not sure though how that would work across all storage? But worth thinking about.

[jennydaman]

I do not perceive any unique challenges supporting this across different storages.

From the plugin instance's point-of-view, which is just a Linux container, it sees a Linux filesystem. This is regardless of what kind of storage is used for CUBE nor k8s.

Suppose CUBE and pfcon are configured to use S3 storage. Currently, pfcon's code is such that symlinks are checked for and silently ignored:

https://github.com/FNNDSC/pfcon/blob/ed00eba024e867b1430645c8c5ce5bb29373b125/pfcon/storage/s3_storage.py#L132

To implement my proposed idea, around this line is where you would add the logic to have a .chrislink be created in place of the symlink.