After full single subject processing, the generation of GROUP level results (report and CSV) is not possible without raw BIDS data present

[FeHoff]

What happened?

Disclaimer: This is a bug for me but maybe a feature request for others: #1387

When all subjects MRIQC results are present locally, the generation of group level results (report and CSV) needs ALL relevant raw data present for BIDS verification. This renders the generation of group stats for larger datasets unnecessarily complicated (as raw data got to be present in one place) and even prevents (bold) group stats creation for very big datasets (i.e. ABCD).

Below is an example for the OpenNeuro NARPS dataset, which is about 270GB in size.

What command did you use?

singularity exec -B $(pwd) --cleanenv code/repronim_containers/images/bids/bids-mriqc--24.0.2.sing /singularity inputs/NARPS MRIQC group --no-datalad-get --notrack --modalities bold --verbose --nprocs 2 --mem 10000 --work-dir /tmp --verbose-reports

What version of the software are you running?

24.0.2

How are you running this software?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

Work directory

Please copy and paste any relevant log output.

2025-03-19 18:50:30 | WARNING  | matplotlib       | Matplotlib created a temporary cache directory at /tmp/matplotlib-88wgiio8 because the default path (/home/mriqc/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+gd5b13cb5.d20240826
  ----------------------------------------------------------------

  NOTICE
  Copyright © The NiPreps Developers.
  
  This product includes software developed by
  the NiPreps Community (https://nipreps.org/).
  
  Portions of this software were developed at the Department of
  Psychology at Stanford University, Stanford, CA, US.
  
  This software contains code ultimately derived from the
  PCP Quality Assessment Protocol (QAP;
  http://preprocessed-connectomes-project.org/quality-assessment-protocol)
  by C. Craddock, S. Giavasis, D. Clark, Z. Shezhad, and J. Pellman.
  
  This software is also distributed as a Docker container image.
  The bootstrapping file for the image ("Dockerfile") is licensed
  under the MIT License.

  ----------------------------------------------------------------

  * BIDS dataset path: /data/project/QC_workflow/DATA/NARPS_QCworkflow/inputs/NARPS.
  * Output folder: MRIQC.
  * Analysis levels: ['group'].
------------------------------------------------------------------
------------------------------------------------------------------

2025-03-19 18:50:32 | IMPORTANT | mriqc            | Extracting metadata and entities for 432 input runs of modality 'bold'...
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 43, in main
    parse_args(argv)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/parser.py", line 658, in parse_args
    initialize_meta_and_data()
  File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 473, in initialize_meta_and_data
    metadata, entities, size, valid = asyncio.run(
                                      ^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 421, in _extract_meta_and_size
    metadata, entities, sizes, valid = list(zip(*await asyncio.gather(*tasks)))
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 75, in worker
    return await loop.run_in_executor(None, job)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/utils/misc.py", line 349, in _file_meta_and_size
    size = os.path.getsize(files) / (1024**3)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 50, in getsize
FileNotFoundError: [Errno 2] No such file or directory: '/data/project/QC_workflow/DATA/NARPS_QCworkflow/inputs/NARPS/sub-001/func/sub-001_task-MGT_run-03_bold.nii.gz'

Additional information / screenshots

The input data is present as Datalad dataset without file content:

> ll /data/project/QC_workflow/DATA/NARPS_QCworkflow/inputs/NARPS/sub-001/func/sub-001_task-MGT_run-03_bold.nii.gz
lrwxrwxrwx 1 fhoffstaedter BnB 144 Dec 23 01:04 /data/project/QC_workflow/DATA/NARPS_QCworkflow/inputs/NARPS/sub-001/func/sub-001_task-MGT_run-03_bold.nii.gz -> ../../.git/annex/objects/VF/QK/MD5E-s689359965--525ba9985ceb4b361fa76b237f802f27.nii.gz/MD5E-s689359965--525ba9985ceb4b361fa76b237f802f27.nii.gz

The MRIQC single subject data is present:

> ll /data/project/QC_workflow/DATA/NARPS_QCworkflow/MRIQC/sub-001
sub-001/                            sub-001_task-MGT_run-01_bold.html@  sub-001_task-MGT_run-03_bold.html@
sub-001_T1w.html@                   sub-001_task-MGT_run-02_bold.html@  sub-001_task-MGT_run-04_bold.html@