Skip to content

[BUG]: Cannot compute markers on AOMIC ID1000 using native space#487

Merged
synchon merged 2 commits intomainfrom
fix/mask-check-native
Mar 3, 2026
Merged

[BUG]: Cannot compute markers on AOMIC ID1000 using native space#487
synchon merged 2 commits intomainfrom
fix/mask-check-native

Conversation

@synchon
Copy link
Copy Markdown
Member

@synchon synchon commented Mar 2, 2026

Is there an existing issue for this?

  • I have searched the existing issues

Current Behavior

I want to compute ReHo/FC in native space. fMRIPrepConfoundRemover fails.

I get this error

2026-02-27 15:54:29 [info     ] Preprocessing data with fMRIPrepConfoundRemover [junifer]
2026-02-27 15:54:29 [info     ] Preprocessing BOLD             [junifer]
2026-02-27 15:54:30 [info     ] No `t_r` specified, using t_r from NIfTI header [junifer]
2026-02-27 15:54:30 [info     ] Read t_r from NIfTI header: 2.200000047683716 [junifer]
Traceback (most recent call last):
  File "/home/fraimondo/miniforge3/envs/junifer/bin/junifer", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/cli/cli.py", line 185, in run
    cli_func.run(
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/api/functions.py", line 223, in run
    mc.fit(datagrabber_object[t_element])
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/pipeline/marker_collection.py", line 103, in fit
    data = preprocessor.fit_transform(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 232, in fit_transform
    return self._fit_transform(input=input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/preprocess/base.py", line 181, in _fit_transform
    t_out = self.preprocess(input=t_input, extra_input=extra_input)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/preprocess/confounds/fmriprep_confound_remover.py", line 640, in preprocess
    mask_img = get_data(
               ^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/data/_dispatch.py", line 160, in get_data
    return registry().get(
           ^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/data/masks/_masks.py", line 487, in get
    set(masks) == {nilearn_callable_masks[0]}
    ^^^^^^^^^^
TypeError: unhashable type: 'CommentedMap'

Expected Behavior

Obtain markers in native space

Steps To Reproduce

  1. With junifer latest
  2. Run any element from the AOMIC ID1000 datasets
workdir: /tmp

datagrabber:
    kind: DataladAOMICID1000
    space: native

preprocess:
  - kind: fMRIPrepConfoundRemover
    detrend: true
    standardize: true
    strategy:
        motion: full
        wm_csf: full
        global_signal: full
    low_pass: 0.08
    high_pass: 0.01
    masks:
      - inherit
      - compute_epi_mask
      - threshold: 0
  - kind: SpaceWarper
    reference: T1w
    on: BOLD
    using: ants

markers:
  - name: ReHo-Julich-Brain_V3_1
    kind: ReHoParcels
    parcellation: Julich-Brain_V3_1
    using: afni
    masks: 
      - inherit

storage:
  kind: HDF5FeatureStorage
  uri: /data/group/riseml/fraimondo/2026_EBRAINS/features/ds003097_ReHo_native/ds003097_ReHo_native.hdf5

queue:
  jobname: aomic_reho_native
  kind: HTCondor
  env:
    kind: conda
    name: junifer
  mem: 8G
  disk: 5G
  verbose: info
  pre_run: |
    source /data/group/appliedml/tools/afni_24.3.06/afni.sh	
    source /data/group/appliedml/tools/ants_2.5.0/ants.sh
    source /data/group/appliedml/tools/fsl_6.0.4-patched2/fsl.sh

Environment

junifer:
  version: 0.0.7.dev320
python:
  version: 3.12.7
  implementation: CPython
dependencies:
  click: 8.1.7
  numpy: 1.26.4
  scipy: 1.14.1
  datalad: 1.1.3
  pandas: 2.2.3
  nibabel: 5.3.2
  nilearn: 0.10.4
  sqlalchemy: 2.0.36
  ruamel.yaml: 0.18.6
  templateflow: 24.2.2
  lazy_loader: '0.4'
  looseversion: None
  junifer_data: None
  structlog: 25.5.0
system:
  platform: Linux-6.12.9+bpo-amd64-x86_64-with-glibc2.36
environment:
  PATH: 
    /home/fraimondo/miniforge3/envs/junifer/bin:/home/fraimondo/.local/bin:/home/fraimondo/miniforge3/condabin:/usr/local/bin:/home/fraimondo/.vscode-server/data/User/globalStorage/github.copilot-chat/debugCommand:/home/fraimondo/.vscode-server/data/User/globalStorage/github.copilot-chat/copilotCli:/home/fraimondo/.vscode-server/cli/servers/Stable-072586267e68ece9a47aa43f8c108e0dcbf44622/server/bin/remote-cli:/home/fraimondo/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/games:/home/fraimondo/.vscode-server/extensions/ms-python.debugpy-2025.18.0-linux-x64/bundled/scripts/noConfigScripts

Relevant log output

2026-02-27 15:50:26 [warning  ] Requested extension 'next' is not available [datalad.support.entrypoints]
2026-02-27 15:50:31 [info     ] ===== Lib Versions =====       [junifer]
2026-02-27 15:50:31 [info     ] click: 8.1.7                   [junifer]
2026-02-27 15:50:31 [info     ] numpy: 1.26.4                  [junifer]
2026-02-27 15:50:31 [info     ] scipy: 1.14.1                  [junifer]
2026-02-27 15:50:31 [info     ] datalad: 1.1.3                 [junifer]
2026-02-27 15:50:31 [info     ] pandas: 2.2.3                  [junifer]
2026-02-27 15:50:31 [info     ] nibabel: 5.3.2                 [junifer]
2026-02-27 15:50:31 [info     ] nilearn: 0.10.4                [junifer]
2026-02-27 15:50:31 [info     ] sqlalchemy: 2.0.36             [junifer]
2026-02-27 15:50:31 [info     ] ruamel.yaml: 0.18.6            [junifer]
2026-02-27 15:50:31 [info     ] templateflow: 24.2.2           [junifer]
2026-02-27 15:50:31 [info     ] junifer_data: None             [junifer]
2026-02-27 15:50:31 [info     ] junifer: 0.0.7.dev320          [junifer]
2026-02-27 15:50:31 [info     ] ========================       [junifer]
2026-02-27 15:50:31 [info     ] Parsing yaml file: /home/fraimondo/dev/events/2026_EBRAINS/1_junifer/junifer_jobs/aomic_reho_native/config.yaml [junifer]
2026-02-27 15:50:31 [info     ] Registering PatternDataGrabber in datagrabber [junifer]
2026-02-27 15:50:31 [info     ] Registering PatternDataladDataGrabber in datagrabber [junifer]
2026-02-27 15:50:31 [info     ] Registering DataladAOMICID1000 in datagrabber [junifer]
2026-02-27 15:50:31 [info     ] `datadir` is None, creating a temporary directory [junifer]
2026-02-27 15:50:31 [info     ] `datadir` set to /tmp/tmpkal3yrx_/dataladu0wt22co/datadir [junifer]
2026-02-27 15:50:31 [info     ] Registering ParcelAggregation in marker [junifer]
2026-02-27 15:50:31 [info     ] Registering ReHoParcels in marker [junifer]
2026-02-27 15:50:31 [info     ] Registering HDF5FeatureStorage in storage [junifer]
2026-02-27 15:50:31 [info     ] Registering fMRIPrepConfoundRemover in preprocessing [junifer]
2026-02-27 15:50:31 [info     ] Registering SpaceWarper in preprocessing [junifer]
2026-02-27 15:50:31 [info     ] Validating Marker Collection   [junifer]
2026-02-27 15:50:31 [info     ] DataGrabber output type: ['BOLD', 'T1w', 'VBM_CSF', 'VBM_GM', 'VBM_WM', 'DWI', 'FreeSurfer', 'Warp'] [junifer]
2026-02-27 15:50:31 [info     ] Validating Data Reader:        [junifer]
2026-02-27 15:50:31 [info     ] Data Reader output type: ['BOLD', 'T1w', 'VBM_CSF', 'VBM_GM', 'VBM_WM', 'DWI', 'FreeSurfer', 'Warp'] [junifer]
2026-02-27 15:50:31 [info     ] Validating Preprocessor: fMRIPrepConfoundRemover [junifer]
2026-02-27 15:50:31 [info     ] Preprocessor input type: ['BOLD', 'T1w', 'VBM_CSF', 'VBM_GM', 'VBM_WM', 'DWI', 'FreeSurfer', 'Warp'] [junifer]
2026-02-27 15:50:31 [info     ] Preprocessor output type: ['VBM_GM', 'DWI', 'Warp', 'VBM_WM', 'T1w', 'VBM_CSF', 'FreeSurfer', 'BOLD'] [junifer]
2026-02-27 15:50:31 [info     ] Validating Preprocessor: SpaceWarper [junifer]
2026-02-27 15:50:31 [info     ] Preprocessor input type: ['VBM_GM', 'DWI', 'Warp', 'VBM_WM', 'T1w', 'VBM_CSF', 'FreeSurfer', 'BOLD'] [junifer]
2026-02-27 15:50:32 [info     ] Preprocessor output type: ['VBM_GM', 'DWI', 'Warp', 'T1w', 'VBM_WM', 'VBM_CSF', 'FreeSurfer', 'BOLD'] [junifer]
2026-02-27 15:50:32 [info     ] Validating Marker: ReHo-Julich-Brain_V3_1 [junifer]
2026-02-27 15:50:33 [info     ] Marker output type: ['vector'] [junifer]
2026-02-27 15:50:33 [info     ] Validating storage for ReHo-Julich-Brain_V3_1 [junifer]
2026-02-27 15:50:33 [info     ] Attempting a clone into /tmp/tmpkal3yrx_/dataladu0wt22co/datadir [datalad.core.distributed.clone] dlm_progress=cloneds dlm_progress_label=Cloning dlm_progress_total=2 dlm_progress_unit=' candidates'
2026-02-27 15:50:33 [info     ] Attempting to clone from https://github.com/OpenNeuroDatasets/ds003097.git to /tmp/tmpkal3yrx_/dataladu0wt22co/datadir [datalad.core.distributed.clone] dlm_progress=cloneds dlm_progress_increment=True dlm_progress_update=1
2026-02-27 15:51:39 [info     ] Completed clone attempts for Dataset(/tmp/tmpkal3yrx_/dataladu0wt22co/datadir) [datalad.core.distributed.clone] dlm_progress=cloneds
2026-02-27 15:51:46 [info     ] access to 1 dataset sibling s3-BACKUP not auto-enabled, enable with:
		datalad siblings -d "/tmp/tmpkal3yrx_/dataladu0wt22co/datadir" enable -s s3-BACKUP [datalad.core.distributed.clone]
2026-02-27 15:51:47 [info     ] Getting element sub-0001       [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for BOLD [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for BOLD.mask [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for BOLD.confounds [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for BOLD.reference [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for T1w [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for T1w.mask [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for VBM_CSF [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for VBM_GM [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for VBM_WM [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for DWI [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer.aseg [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer.norm [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer.lh_white [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer.rh_white [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer.lh_pial [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for FreeSurfer.rh_pial [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for Warp.0 [junifer]
2026-02-27 15:51:47 [info     ] Resolving path from pattern for Warp.1 [junifer]
2026-02-27 15:54:29 [info     ] Fitting pipeline               [junifer]
2026-02-27 15:54:29 [info     ] Reading BOLD.mask from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/func/sub-0001_task-moviewatching_space-T1w_desc-brain_mask.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] BOLD.mask is of type NIFTI     [junifer]
2026-02-27 15:54:29 [info     ] Reading BOLD.confounds from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/func/sub-0001_task-moviewatching_desc-confounds_regressors.tsv [junifer]
2026-02-27 15:54:29 [info     ] BOLD.confounds is of type TSV  [junifer]
2026-02-27 15:54:29 [info     ] Reading BOLD.reference from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/func/sub-0001_task-moviewatching_space-T1w_boldref.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] BOLD.reference is of type NIFTI [junifer]
2026-02-27 15:54:29 [info     ] Reading BOLD from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/func/sub-0001_task-moviewatching_space-T1w_desc-preproc_bold.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] BOLD is of type NIFTI          [junifer]
2026-02-27 15:54:29 [info     ] Reading T1w.mask from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/anat/sub-0001_desc-brain_mask.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] T1w.mask is of type NIFTI      [junifer]
2026-02-27 15:54:29 [info     ] Reading T1w from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/anat/sub-0001_desc-preproc_T1w.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] T1w is of type NIFTI           [junifer]
2026-02-27 15:54:29 [info     ] Reading VBM_CSF from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/anat/sub-0001_label-CSF_probseg.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] VBM_CSF is of type NIFTI       [junifer]
2026-02-27 15:54:29 [info     ] Reading VBM_GM from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/anat/sub-0001_label-GM_probseg.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] VBM_GM is of type NIFTI        [junifer]
2026-02-27 15:54:29 [info     ] Reading VBM_WM from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/fmriprep/sub-0001/anat/sub-0001_label-WM_probseg.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] VBM_WM is of type NIFTI        [junifer]
2026-02-27 15:54:29 [info     ] Reading DWI from /tmp/tmpkal3yrx_/dataladu0wt22co/datadir/derivatives/dwipreproc/sub-0001/dwi/sub-0001_desc-preproc_dwi.nii.gz [junifer]
2026-02-27 15:54:29 [info     ] DWI is of type NIFTI           [junifer]
2026-02-27 15:54:29 [info     ] Preprocessing data with fMRIPrepConfoundRemover [junifer]
2026-02-27 15:54:29 [info     ] Preprocessing BOLD             [junifer]
2026-02-27 15:54:30 [info     ] No `t_r` specified, using t_r from NIfTI header [junifer]
2026-02-27 15:54:30 [info     ] Read t_r from NIfTI header: 2.200000047683716 [junifer]
Traceback (most recent call last):
  File "/home/fraimondo/miniforge3/envs/junifer/bin/junifer", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/cli/cli.py", line 185, in run
    cli_func.run(
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/api/functions.py", line 223, in run
    mc.fit(datagrabber_object[t_element])
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/pipeline/marker_collection.py", line 103, in fit
    data = preprocessor.fit_transform(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/pipeline/pipeline_step_mixin.py", line 232, in fit_transform
    return self._fit_transform(input=input, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/preprocess/base.py", line 181, in _fit_transform
    t_out = self.preprocess(input=t_input, extra_input=extra_input)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/preprocess/confounds/fmriprep_confound_remover.py", line 640, in preprocess
    mask_img = get_data(
               ^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/data/_dispatch.py", line 160, in get_data
    return registry().get(
           ^^^^^^^^^^^^^^^
  File "/home/fraimondo/miniforge3/envs/junifer/lib/python3.12/site-packages/junifer/data/masks/_masks.py", line 487, in get
    set(masks) == {nilearn_callable_masks[0]}
    ^^^^^^^^^^
TypeError: unhashable type: 'CommentedMap'

Anything else?

No response

@fraimondo fraimondo added bug Something isn't working triage New issues waiting to be reviewed labels Mar 1, 2026
@synchon synchon added mask Issues or pull requests related to masks and removed triage New issues waiting to be reviewed labels Mar 2, 2026
@synchon
Copy link
Copy Markdown
Member

synchon commented Mar 2, 2026

@fraimondo Can you check if this branch solves your issue? Also you are working with native space data from the beginning, the SpaceWarper is likely not needed.

@synchon synchon added this to the 0.0.7 (alpha 6) milestone Mar 2, 2026
@synchon synchon merged commit 6e665b1 into main Mar 3, 2026
2 of 8 checks passed
@synchon synchon deleted the fix/mask-check-native branch March 3, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working mask Issues or pull requests related to masks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants