Nuisance crash with NHP pipeline

[sgiavasis]

No description provided.

[beruss]

I am attempting to run the default NHP pipeline on a single resting state session from one subject. Currently using the Yerkes19 template as the template space. It started working well but eventually crashed. Below is the output from when the crash occurred. The cerebrospinal fluid nuisance regressor appears to be missing. This is evidenced by the /working.../CerebrospinalFluid_union_masks/csf_lat_ven_mask_calc_mask.nii.gz and /working/.../VentriclesToAnat_Functional_ants/csf_mask_xtedit_trans.nii.gz files being empty in the nuisance_0_0 folders.
Additionally, the while the FAST segmentation function is working on the T1w image, the results are poor due to noise in the T1w image, mostly likely.

[pn2200]

I think that I have the same issue, where the cerebrospinal fluid nuisance regressor is missing, eventually causing the following error, which prevents the pipeline from completing:

200113-14:31:15,22 nipype.workflow ERROR:
	 Node build_nuisance_regressors.a0.b09 failed to run on host n222.
200113-14:31:15,22 nipype.workflow ERROR:
	 Saving crash info to /output/crash/crash-20200113-143115-pnovak2-build_nuisance_regressors.a0.b09-3cfbb012-b5d4-4133-b6aa-6e500bc4d212.pklz
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 158, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/nuisance/nuisance.py", line 164, in gather_nuisance
    regressor_length))
ValueError: Number of time points in /output/working/resting_preproc_sub-DEV127_ses-wave2/nuisance_0_0/_scan_rest_acq-2/_selector_CSF-2mmE-M_aC-WM-2mm-DPC5_G-M_M-SDB_P-2/CerebrospinalFluid_mean/roi_stats.csv (0) is inconsistent with length of functional file /output/working/resting_preproc_sub-DEV127_ses-wave2/func_preproc_afni_0/_scan_rest_acq-2/func_normalize/sub-DEV127_ses-wave2_task-rest_acq-2_bold_calc_tshift_resample_volreg_calc_maths.nii.gz (414)

Can you help resolve this issue?

[HechengJin0]

It is general issue when refine csf mask with template ventricle mask in nuisance step. For some data, registration of template ventricle mask to nuisance space(individual anat OR functional space) is off, it makes refined csf mask (individual csf * template ventricle mask) incorrect, the csf mask might become blank(another word, number of time points is 0). so it crashes at nuisance regression step.
In this case, it would be good to try without ventricle mask. We have added toggle for ventricle mask refinement in release 1.6.1, which is upcoming in few days.
Attached is example of template ventricle mask in individual space.

[pn2200]

If I understand correctly, this is an issue that occurs for some subjects, but not necessarily for all subjects. Is there a way to determine that this will happen beforehand? I have run C-PAC on 10 subjects, and each one has displayed this issue. Is there some way to avoid it in the current (1.6.0) release?

Also, what tool are you using to display the images below? Are you visualizing output from C-PAC? If so, which files are you inspecting?

[HechengJin0]

Yes, it depends on data. Currently, we don't have a good solution if it crashes due to blank csf mask. You could try v. 1.6.1, it will release soon. What kind of data are you running? Human or Primes?
Usually, I use fslview or fsleyes to visualize the output.

[pn2200]

In the release notes for version 1.6.1, it mentions toggling CSF regressor:

CSF Regressor Configurability. Ventricle mask refinement of CSF masks for CSF-based nuisance regression can now be toggled.

What is the name of this configuration item? I looked briefly in the default_pipeline.yml and didn't see an obvious configuration item, and I can't find any further documentation on this feature.

[HechengJin0]

@pn2200 Sorry for the confusion. You could set lateral_ventricles_mask field as None in pipeline config. Please check attached link. lateral_ventricles_mask : None https://github.com/FCP-INDI/C-PAC/blob/master/dev/docker_data/default_pipeline.yml#L740

[pn2200]

I think I am still having the same crash when disabling the lateral_ventricles_mask. The error reports that the CSF_mean nuisance regressor has a number of time points inconsistent with the length of the functional file:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 158, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/nuisance/nuisance.py", line 167, in gather_nuisance
    regressor_length))
ValueError: Number of time points in /output/working/resting_preproc_sub-DEV107_ses-wave1/nuisance_regressor_1_0/_scan_ROC_acq-2/_selector_CSF-2mmE-M_aC-WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/CerebrospinalFluid_mean/roi_stats.csv (0) is inconsistent with length of functional file /output/working/resting_preproc_sub-DEV107_ses-wave1/func_preproc_afni_0/_scan_ROC_acq-2/func_normalize/sub-DEV107_ses-wave1_task-ROC_acq-2_bold_calc_tshift_resample_volreg_calc_maths.nii.gz (141)

Previously I was using the built-in Harvard-Oxford atlas; when I look at the overlay of the ventricle mask over the individual, it doesn't look obviously wrong:

Do you have any other suggestion on how to avoid this issue?

[anibalsolon]

@pn2200 I would try to disable erosion, as it might be removing all the voxels from your CSF mask.

   CerebrospinalFluid:
     summary: Mean
     extraction_resolution: 2
     erode_mask: false

[shnizzedy]

Related:

• CPAC error when its running nuisance.py
• Build_nuisance_regressor crash: number of time points is inconsistent