fMRI nipype example workflow not working

[TheChymera]

Hi, I'm new to nipype and I'd like to see it in action before I decide in how far I should use it. I am using the live versions of nipype, nipy, nibabel etc. and I am trying to run the SPM famous vs. non-famous faces example workflow.

I am getting this error. I see there are some crash files it saves but I can't really open them - gedit does show me some of the text they contain and it seems it's a python script :-/

Could you help me out?

/usr/lib64/python2.7/site-packages/nipy/labs/glm/glm.py:7: FutureWarning: Module nipy.labs.utils.routines deprecated, will be removed
  from ..utils import mahalanobis
/usr/lib64/python2.7/site-packages/nipype/interfaces/nipy/model.py:17: FutureWarning: Module nipy.labs.glm deprecated, will be removed. Please use nipy.modalities.fmri.glm instead.
  import nipy.labs.glm.glm as GLM
140125-03:52:23,596 workflow INFO:
     ['check', 'execution', 'logging']
140125-03:52:23,857 workflow INFO:
     Running serially.
140125-03:52:23,861 workflow INFO:
     Executing node datasource.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/_subject_id_M03953/datasource
140125-03:52:23,946 workflow ERROR:
     ['Node datasource.a0 failed to run on host localhost.']
140125-03:52:23,952 workflow INFO:
     Saving crash info to /home/chymera/src/faceOM/analysis/crash-20140125-035223-chymera-datasource.a0.npz
140125-03:52:23,952 workflow INFO:
     Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/linear.py", line 38, in run
    node.run(updatehash=updatehash)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 1386, in run
    self._run_interface()
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 1496, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 1622, in _run_command
    result = self._interface.run()
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 931, in run
    self._check_mandatory_inputs()
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 869, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: DataGrabber requires a value for input 'sort_filelist'. For a list of required inputs, see DataGrabber.help()

140125-03:52:23,960 workflow INFO:
     ***********************************
140125-03:52:23,960 workflow ERROR:
     could not run node: level1.datasource.a0
140125-03:52:23,960 workflow INFO:
     crashfile: /home/chymera/src/faceOM/analysis/crash-20140125-035223-chymera-datasource.a0.npz
140125-03:52:23,961 workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "fmri.py", line 486, in <module>
    level1.run()
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 689, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/linear.py", line 56, in run
    report_nodes_not_run(notrun)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 90, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details

[satra]

a couple of things:

• you can display the crash files with nipype_display_crash which should be installed when you install nipype
• are you sure you have the latest release (0.9.1) or the current master from github. (this seems to run fine on my machine)

check with: python -c "import nipype; print nipype.__version__"

and then if the version is proper run the following:

mkdir spm_face_data
cd spm_face_data
wget http://www.fil.ion.ucl.ac.uk/spm/download/data/face_rep/rawepi.zip
unzip rawepi.zip 
wget http://www.fil.ion.ucl.ac.uk/spm/download/data/face_rep/structural.zip
unzip structural.zip
wget http://www.fil.ion.ucl.ac.uk/spm/download/data/face_rep/sots.mat
wget http://www.fil.ion.ucl.ac.uk/spm/download/data/face_rep/all-conditions.mat
cd ..
python /path/to/nipype/examples/fmri_spm_face.py

[TheChymera]

$ python -c "import nipype; print nipype.__version__"
/usr/lib64/python2.7/site-packages/nipy/labs/glm/glm.py:7: FutureWarning: Module nipy.labs.utils.routines deprecated, will be removed
  from ..utils import mahalanobis
/usr/lib64/python2.7/site-packages/nipype/interfaces/nipy/model.py:17: FutureWarning: Module nipy.labs.glm deprecated, will be removed. Please use nipy.modalities.fmri.glm instead.
  import nipy.labs.glm.glm as GLM
0.9.1

As for the rest of what you said - can I not run the examples from another directory? I am still getting the exact same error after doing everything you recommended sans the last line (I keep my data analysis scripts in a special directory, and if I'll be using nipy it will in nay case be from that dir.

Also, I did two minor modifications to the workflow (which I should perhaps have mentioned earlier):

• line 225: data_dir = os.path.abspath("/home/chymera/data/famous_faces")
• line 332: slice_timingref.ref_slice = int(num_slices/2)

[satra]

the reason it's puzzling is that the error you are getting in the post above is about datagrabber and that definitely has sort_filelist set in the example.

https://github.com/nipy/nipype/blob/master/examples/fmri_spm_face.py#L257

could you do a diff between your file and the one in the nipype source directory?

[TheChymera]

Ok, the code I had was missing datasource.inputs.sort_filelist = True, and had an additional:

#!/usr/bin/env python
from __future__ import division

(which I always add to my scripts because it makes sense! :P)

Now I added that one line (while not deleting my custom preamble) and I get this slightly different error:

/usr/lib64/python2.7/site-packages/nipy/labs/glm/glm.py:7: FutureWarning: Module nipy.labs.utils.routines deprecated, will be removed
  from ..utils import mahalanobis
/usr/lib64/python2.7/site-packages/nipype/interfaces/nipy/model.py:17: FutureWarning: Module nipy.labs.glm deprecated, will be removed. Please use nipy.modalities.fmri.glm instead.
  import nipy.labs.glm.glm as GLM
140126-21:56:14,600 workflow INFO:
     ['check', 'execution', 'logging']
140126-21:56:14,787 workflow INFO:
     Running serially.
140126-21:56:14,789 workflow INFO:
     Executing node datasource.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/_subject_id_M03953/datasource
140126-21:56:15,281 workflow INFO:
     Executing node realign.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/realign
140126-22:01:08,613 workflow INFO:
     Executing node coregister.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/coregister
140126-22:02:00,390 workflow INFO:
     Executing node segment.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/segment
140126-22:06:01,976 workflow INFO:
     Executing node normalize_struc.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/normalize_struc
140126-22:06:26,899 workflow ERROR:
     ['Node normalize_struc.a0 failed to run on host localhost.']
140126-22:06:26,915 workflow INFO:
     Saving crash info to /home/chymera/src/faceOM/analysis/crash-20140126-220626-chymera-normalize_struc.a0.npz
140126-22:06:26,915 workflow INFO:
     Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/linear.py", line 38, in run
    node.run(updatehash=updatehash)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 1386, in run
    self._run_interface()
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 1496, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 1622, in _run_command
    result = self._interface.run()
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 946, in run
    runtime = self._run_interface(runtime)
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/spm/base.py", line 295, in _run_interface
    results = self.mlab.run()
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 946, in run
    runtime = self._run_interface(runtime)
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/matlab.py", line 146, in _run_interface
    self.raise_exception(runtime)
  File "/usr/lib64/python2.7/site-packages/nipype/interfaces/base.py", line 1313, in raise_exception
    raise RuntimeError(message)
RuntimeError: Command:
matlab -nodesktop -nosplash -nodesktop -nosplash -singleCompThread -r "addpath('/home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/normalize_struc');pyscript_normalize;exit"
Standard output:
Warning: Unable to open display ':1'.  You will not be able to display graphics on the screen.
Warning: No window system found.  Java option 'MWT' ignored.

                                                                                                     < M A T L A B (R) >
                                                                                           Copyright 1984-2013 The MathWorks, Inc.
                                                                                             R2013a (8.1.0.604) 64-bit (glnxa64)
                                                                                                      February 15, 2013


To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.

Executing pyscript_normalize at 26-Jan-2014 22:06:15:
-------------------------------------------------------------------------------------------------------
MATLAB Version: 8.1.0.604 (R2013a)
MATLAB License Number: 724504
Operating System: Linux 3.12.5-gentoo #1 SMP Thu Dec 26 05:15:20 EET 2013 x86_64
Java Version: Java 1.6.0_17-b04 with Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server VM mixed mode
-------------------------------------------------------------------------------------------------------
MATLAB                                                Version 8.1        (R2013a)
Simulink                                              Version 8.1        (R2013a)
Aerospace Blockset                                    Version 3.11       (R2013a)
Aerospace Toolbox                                     Version 2.11       (R2013a)
Bioinformatics Toolbox                                Version 4.3        (R2013a)
Communications System Toolbox                         Version 5.4        (R2013a)
Computer Vision System Toolbox                        Version 5.2        (R2013a)
Control System Toolbox                                Version 9.5        (R2013a)
Curve Fitting Toolbox                                 Version 3.3.1      (R2013a)
DO Qualification Kit                                  Version 2.1        (R2013a)
DSP System Toolbox                                    Version 8.4        (R2013a)
Database Toolbox                                      Version 4.1        (R2013a)
Datafeed Toolbox                                      Version 4.5        (R2013a)
Econometrics Toolbox                                  Version 2.3        (R2013a)
Embedded Coder                                        Version 6.4        (R2013a)
Filter Design HDL Coder                               Version 2.9.3      (R2013a)
Financial Instruments Toolbox                         Version 1.1        (R2013a)
Financial Toolbox                                     Version 5.1        (R2013a)
Fixed-Point Designer                                  Version 4.0        (R2013a)
Fuzzy Logic Toolbox                                   Version 2.2.17     (R2013a)
Global Optimization Toolbox                           Version 3.2.3      (R2013a)
HDL Coder                                             Version 3.2        (R2013a)
HDL Verifier                                          Version 4.2        (R2013a)
IEC Certification Kit                                 Version 3.1        (R2013a)
Image Acquisition Toolbox                             Version 4.5        (R2013a)
Image Processing Toolbox                              Version 8.2        (R2013a)
Instrument Control Toolbox                            Version 3.3        (R2013a)
MATLAB Builder JA                                     Version 2.2.6      (R2013a)
MATLAB Coder                                          Version 2.4        (R2013a)
MATLAB Compiler                                       Version 4.18.1     (R2013a)
MATLAB Report Generator                               Version 3.14       (R2013a)
Mapping Toolbox                                       Version 3.7        (R2013a)
Model Predictive Control Toolbox                      Version 4.1.2      (R2013a)
Neural Network Toolbox                                Version 8.0.1      (R2013a)
Optimization Toolbox                                  Version 6.3        (R2013a)
Parallel Computing Toolbox                            Version 6.2        (R2013a)
Partial Differential Equation Toolbox                 Version 1.2        (R2013a)
Phased Array System Toolbox                           Version 2.0        (R2013a)
RF Toolbox                                            Version 2.12       (R2013a)
Robust Control Toolbox                                Version 4.3        (R2013a)
Signal Processing Toolbox                             Version 6.19       (R2013a)
SimBiology                                            Version 4.3        (R2013a)
SimDriveline                                          Version 2.4        (R2013a)
SimElectronics                                        Version 2.3        (R2013a)
SimEvents                                             Version 4.3        (R2013a)
SimHydraulics                                         Version 1.12       (R2013a)
SimMechanics                                          Version 4.2        (R2013a)
SimPowerSystems                                       Version 5.8        (R2013a)
SimRF                                                 Version 4.0        (R2013a)
Simscape                                              Version 3.9        (R2013a)
Simulink 3D Animation                                 Version 6.3        (R2013a)
Simulink Code Inspector                               Version 1.3        (R2013a)
Simulink Coder                                        Version 8.4        (R2013a)
Simulink Control Design                               Version 3.7        (R2013a)
Simulink Design Optimization                          Version 2.3        (R2013a)
Simulink Design Verifier                              Version 2.4        (R2013a)
Simulink Report Generator                             Version 3.14       (R2013a)
Simulink Verification and Validation                  Version 3.5        (R2013a)
Stateflow                                             Version 8.1        (R2013a)
Statistical Parametric Mapping                        Version 5236       (SPM8)  
Statistics Toolbox                                    Version 8.2        (R2013a)
Symbolic Math Toolbox                                 Version 5.10       (R2013a)
System Identification Toolbox                         Version 8.2        (R2013a)
SystemTest                                            Version 2.6.5      (R2013a)
Wavelet Toolbox                                       Version 4.11       (R2013a)
SPM version: SPM8 Release: 5236
SPM path: /home/chymera/src/spm8/spm.m


------------------------------------------------------------------------
Running job #1
------------------------------------------------------------------------
No executable modules, but still unresolved dependencies or incomplete module inputs.
The following modules did not run:
Skipped: Normalise: Write


Standard error:
No window system found.  Java option 'MWT' ignored.
MATLAB code threw an exception:
Job execution failed. The full log of this run can be found in MATLAB command window, starting with the lines (look for the line showing the exact #job as displayed in this error message)
------------------ 
Running job #1
------------------

File:cfg_util.m
Name:spm_jobman.m
Line:112
File:yscript_normalize.m
Name:cfg_util
Line:115
File:pm_jobman
Name:pyscript_normalize
Line:835
File:Ù
Name:�
Line:
Return code: 0
Interface MatlabCommand failed to run. 
Interface Normalize failed to run. 

140126-22:06:27,27 workflow INFO:
     Executing node slice_timing.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/slice_timing
140126-22:08:02,146 workflow INFO:
     Executing node normalize_func.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/normalize_func
140126-22:09:07,627 workflow INFO:
     Executing node smooth.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/preproc/_subject_id_M03953/smooth
140126-22:10:48,214 workflow INFO:
     Executing node modelspec.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/analysis/_subject_id_M03953/modelspec
140126-22:10:48,904 workflow INFO:
     Executing node modelspec.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/paramanalysis/_subject_id_M03953/modelspec
140126-22:10:49,686 workflow INFO:
     Executing node level1design.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/paramanalysis/_subject_id_M03953/level1design
140126-22:11:32,376 workflow INFO:
     Executing node level1estimate.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/paramanalysis/_subject_id_M03953/level1estimate
140126-22:12:36,870 workflow INFO:
     Executing node contrastestimate.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/paramanalysis/_subject_id_M03953/contrastestimate
140126-22:13:13,287 workflow INFO:
     Executing node threshold.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/paramanalysis/_subject_id_M03953/threshold
140126-22:13:30,350 workflow INFO:
     Executing node level1design.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/analysis/_subject_id_M03953/level1design
140126-22:14:12,429 workflow INFO:
     Executing node level1estimate.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/analysis/_subject_id_M03953/level1estimate
140126-22:15:17,126 workflow INFO:
     Executing node contrastestimate.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/analysis/_subject_id_M03953/contrastestimate
140126-22:15:59,303 workflow INFO:
     Executing node threshold.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/firstlevel/analysis/_subject_id_M03953/threshold
140126-22:16:16,417 workflow INFO:
     Executing node datasink.a0 in dir: /home/chymera/src/faceOM/analysis/spm_face_tutorial/workingdir/level1/_subject_id_M03953/datasink
140126-22:16:16,558 workflow INFO:
     ***********************************
140126-22:16:16,558 workflow ERROR:
     could not run node: level1.firstlevel.preproc.normalize_struc.a0
140126-22:16:16,558 workflow INFO:
     crashfile: /home/chymera/src/faceOM/analysis/crash-20140126-220626-chymera-normalize_struc.a0.npz
140126-22:16:16,558 workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "fmri.py", line 486, in <module>
    level1.run()
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/engine.py", line 689, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/linear.py", line 56, in run
    report_nodes_not_run(notrun)
  File "/usr/lib64/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 90, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details

[satra]

here are the changes required

--- a/software/nipy-repo/masternipype/examples/fmri_spm_face.py
+++ b/fmri_spm_face.py
@@ -80,6 +80,7 @@ coregister.inputs.jobtype = 'estimate'


 segment = pe.Node(interface=spm.Segment(), name="segment")
+segment.inputs.save_bias_corrected = True

 """Uncomment the following line for faster execution
 """
@@ -127,8 +128,8 @@ preproc.connect([(realign,coregister,[('mean_image', 'target')]),
                  (coregister, segment,[('coregistered_source','data')]),
                  (segment, normalize_func, [('transformation_mat','parameter_file')]),
                  (segment, normalize_struc, [('transformation_mat','parameter_file'),
-                                             ('modulated_input_image', 'apply_to_files'),
-                                             (('modulated_input_image', get_vox_dims), 'write_voxe
+                                             ('bias_corrected_image', 'apply_to_files'),
+                                             (('bias_corrected_image', get_vox_dims), 'write_voxel
                  (realign, slice_timing, [('realigned_files', 'in_files')]),
                  (slice_timing, normalize_func, [('timecorrected_files', 'apply_to_files'),
                                             (('timecorrected_files', get_vox_dims), 'write_voxel_s

[TheChymera]

You should also add this correction slice_timingref.ref_slice = int(num_slices/2) - it needs to be an int, so best make it an int.

[satra]

@TheChymera - it's true that we don't respect python 3 conventions that you are catching because of your preamble. i've made this change - but the port to py3k will take some effort.