IndexError in the initial steps of the participant level when saving the config file #1264

oesteban · 2024-04-12T06:47:41Z

(forked from #1087)

First post

Originally posted by @DVSneuro in #1087 (comment)

I'm getting the same IndexError with the latest pre-release for 24.0.0, but maybe it's a conflicting environment issue? I'm using --cleanenv so I thought that option would prevent that, but I am likely misunderstanding something.

Using the same data and code, I'm not getting any errors with 23.1.0. Maybe one notable feature of my dataset is that it is multiecho, and it seems like the handling for multiecho data has improved since 23.1.0. The IndexError appears starting with 23.1.1.

Happy to share data if it's helpful. It's not quite ready for OpenNeuro and public sharing, but maybe I send a few subjects privately via OneDrive or DropBox.

Thanks for any help!

Best wishes,
David

Here is my code:

TEMPLATEFLOW_DIR=/ZPOOL/data/tools/templateflow
export APPTAINERENV_TEMPLATEFLOW_HOME=/opt/templateflow
/ZPOOL/data/tools/apptainer/bin/singularity run --cleanenv \
-B ${TEMPLATEFLOW_DIR}:/opt/templateflow \
-B $maindir:/base \
-B $scratchdir:/scratch \
/ZPOOL/data/tools/mriqc-24.0.0rc2.simg \
/base/bids /base/derivatives/mriqc \
participant --participant_label $sub \
--bids-filter-file /base/code/mriqc_config.json \
-w /scratch

Here is the contents of my mriqc_config.json file:

{
  "bold": {"datatype": "func", "suffix": "bold", "part": [null, "mag"]}
}

And here is the full output of the error:

------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
  ----------------------------------------------------------------
  * BIDS dataset path: /base/bids.
  * Output folder: /base/derivatives/mriqc.
  * Analysis levels: ['participant'].
  * BIDS filters-file: /base/code/mriqc_config.json.------------------------------------------------------------------

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 45, in build_workflow
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 123, in main
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range

Follow up post

Originally posted by @DVSneuro in #1087 (comment)

Thanks, @oesteban -- sorry, I wasn't sure if I should create a new issue for this error. I do get the same error with the 24.0.0rc2 version. I can also confirm that it happens without the config file being used.

I got a different error with a different subject (see below). So, just as a sanity check, I tripled checked the bids validator, and there are no errors there.

Same error, latest pre-release:

bash mriqc.sh 10003
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
  ----------------------------------------------------------------
  * BIDS dataset path: /base/bids.
  * Output folder: /base/derivatives/mriqc.
  * Analysis levels: ['participant'].
  * BIDS filters-file: /base/code/mriqc_config.json.------------------------------------------------------------------

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/workflow.py", line 45, in build_workflow
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 123, in main
    config.load(config_file)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/config.py", line 633, in load
    settings = loads(filename.read_text())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1002, in load_array
    a[b] = a[b] + ',' + a[b + 1]
                        ~^^^^^^^
IndexError: list index out of range

Different subject, new error:

------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
  ----------------------------------------------------------------
  * BIDS dataset path: /base/bids.
  * Output folder: /base/derivatives/mriqc.
  * Analysis levels: ['participant'].
------------------------------------------------------------------

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 511, in loads
    ret = decoder.load_line(line, currentlevel, multikey,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 778, in load_line
    value, vtype = self.load_value(pair[1], strictly_valid)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 880, in load_value
    return (self.load_array(v), "array")
            ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/toml/decoder.py", line 1029, in load_array
    raise ValueError("Not a homogeneous array")
ValueError: Not a homogeneous array

Data

@DVSneuro posted a MWE on openneuro: https://openneuro.org/datasets/ds005085

The text was updated successfully, but these errors were encountered:

oesteban · 2024-04-12T07:15:06Z

Okay let's dissect this:

Environment leaking: it does look like your environment is not well isolated because in both runs (first post and follow-up) you mention you are using RCs, but the initial banner says otherwise:
```
 ------------------------------------------------------------------
   Running MRIQC version 24.1.0.dev0+g9b5c2f63.d20240408
   ----------------------------------------------------------------
   * BIDS dataset path: /base/bids.
   * Output folder: /base/derivatives/mriqc.
   * Analysis levels: ['participant'].
 ------------------------------------------------------------------
```
Indeed, the stated version is 24.1.0.dev0+g9b5c2f63.d20240408 in both runs when it should be 24.0.0rc1 and 24.0.0rc2. Since you have the same version on both, I'm unsure whether you are executing different versions. However, this does not seem to explain the error.
BIDS-filter file: Then I thought the choke point was the BIDS-filter file. In particular, it seems that having "part": [null, "mag"] could be the culprit, in collaboration with this problem of toml: mixed types in array don't work uiri/toml#270. Looks like your filter file is trying to (i) select functional data only and (ii) include only magnitude files (either defining part- or a single file without phase). You can totally remove the BIDS-filter, and add -m bold to your command line.
What the heck?: I was happily believing I had discovered the problem, but then your "Different subject, new error" traceback shows that you ran without bids-filter file.

Before replicating myself, can I ask you for a last piece of information? Can you please copy and paste the contents of your config file? The file should be named something like: $scratchdir/.mriqc.<some-random-string>.toml

DVSneuro · 2024-04-12T11:49:53Z

Thanks, @oesteban -- I'm also confused by the version number that is MRIQC outputs, and I've noticed that discrepancy for a few versions now. I'm not sure what it means for the environment to be leaking. For reference, here's how I built the container:

singularity build mriqc-24.0.0rc6.simg docker://nipreps/mriqc:24.0.0rc6

Also, here's the contents of the config file in scratch directory:

[environment]
cpu_count = 48
exec_env = "singularity"
free_mem = 36.5
freesurfer_home = "/opt/freesurfer"
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.8.6"
synthstrip_path = "PosixPath('/opt/freesurfer/models/synthstrip.1.pt')"
templateflow_version = "24.2.0"
total_memory = 125.58418655395508
version = "24.1.0.dev0+g4d90e96b.d20240411"

[execution]
ants_float = false
bids_dir = "/base"
bids_database_dir = "/scratch/.bids_db-20240412-004639_e02641df-e373-46bb-b565-f8ba543e5367"
bids_database_wipe = false
cwd = "/home/tug87422@tu.temple.edu"
debug = false
dry_run = false
dsname = "<unset>"
float32 = true
layout = "BIDS Layout: /base"
log_dir = "/base/derivatives/mriqc/logs"
log_level = 25
modalities = [ "T1w", "T2w", "bold", "dwi",]
no_sub = false
notrack = false
output_dir = "/base/derivatives/mriqc"
participant_label = [ "10043",]
pdb = false
reports_only = false
resource_monitor = false
run_uuid = "20240412-004639_e02641df-e373-46bb-b565-f8ba543e5367"
templateflow_home = "/templateflow"
upload_strict = false
verbose_reports = false
webapi_url = "https://mriqc.nimh.nih.gov:443/api/v1"
work_dir = "/scratch"
write_graph = false

[workflow]
analysis_level = [ "participant",]
biggest_file_gb = 2.648588978346189e-10
deoblique = false
despike = false
fd_thres = 0.2
fd_radius = 50
fft_spikes_detector = false
min_len_dwi = 7
min_len_bold = 5
species = "human"
template_id = "MNI152NLin2009cAsym"

[nipype]
crashfile_format = "txt"
get_linked_libs = false
local_hash_check = true
nprocs = 48
omp_nthreads = 1
plugin = "MultiProc"
remove_node_directories = false
resource_monitor = false
stop_on_first_crash = true

[settings]
file_path = "/scratch/config-20240412-004639_e02641df-e373-46bb-b565-f8ba543e5367.toml"
start_time = 1712897199.605464

[execution.bids_filters]

[workflow.inputs]
t1w = [ "/base/sub-10043/anat/sub-10043_T1w.nii.gz",]
bold = [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me1_bold.nii.gz", [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-1_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-2_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-3_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb1me4_echo-4_part-mag_bold.nii.gz",], "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me1_bold.nii.gz", [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-1_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-2_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-3_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb3me4_echo-4_part-mag_bold.nii.gz",], "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me1_bold.nii.gz", [ "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-1_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-2_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-3_part-mag_bold.nii.gz", "/base/sub-10043/func/sub-10043_task-sharedreward_acq-mb6me4_echo-4_part-mag_bold.nii.gz",],]

[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

oesteban · 2024-04-16T21:19:11Z

I suspect this comes from mixing multi-echo and single-echo inputs - I'll check tomorrow and submit a fix.

DVSneuro · 2024-04-17T00:41:13Z

Thanks, Oscar! That sounds like a possibility given where it seems to break in these data (e.g., sub-10015 doesn't have multiecho and MRIQC works).

oesteban · 2024-04-17T07:36:39Z

Confirmed - the problem is not because some runs are multi-echo. The problem comes from having both (single-echo and multi-echo) in a single run.

This happened, e.g., when trying to run MRIQC on datasets with both single-echo and multi-echo runs or with filters where some array had mixed types (e.g., `part: [null, 'mag']`). Resolves: #1264.

oesteban mentioned this issue Apr 12, 2024

IndexError: list index out of range (MRIQC never runs longer than 25s) #1087

Closed

oesteban added the bug label Apr 12, 2024

oesteban added this to the 24.0.0 milestone Apr 17, 2024

This was referenced Apr 17, 2024

FIX: Bug in *toml* loader crashing with mixed arrays in config #1281

Merged

Config file loading crashes when some array has mixed types nipreps/fmriprep#3274

Open

oesteban closed this as completed in #1281 Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IndexError in the initial steps of the participant level when saving the config file #1264

IndexError in the initial steps of the participant level when saving the config file #1264

oesteban commented Apr 12, 2024 •

edited

oesteban commented Apr 12, 2024

DVSneuro commented Apr 12, 2024

oesteban commented Apr 16, 2024

DVSneuro commented Apr 17, 2024

oesteban commented Apr 17, 2024

IndexError in the initial steps of the participant level when saving the config file #1264

IndexError in the initial steps of the participant level when saving the config file #1264

Comments

oesteban commented Apr 12, 2024 • edited

First post

Follow up post

Data

oesteban commented Apr 12, 2024

DVSneuro commented Apr 12, 2024

oesteban commented Apr 16, 2024

DVSneuro commented Apr 17, 2024

oesteban commented Apr 17, 2024

oesteban commented Apr 12, 2024 •

edited