Setting mask image fails SPM Level1Design if input 'spm_mat_dir' is not default

[jAchtzehn]

Summary

Actually there are two issues caused by the same problem:

1. issue: When using a mask image for SPM level1design and also changing the input "spm_mat_dir" the Level1Design node fails.

2. issue: Even without changing the spm_mat_dir input, a problem arises when running different analyses in parallel: because each analysis is loading/writing the same SPM.mat file (in the default MATLAB folder) a file I/O error is happening at random (after ~50 runs).

Actual behavior

During level1design with an explicit mask as an input, the script is trying to load a (temporary?!) SPM.mat file in the folder (i) specified by "spm_mat_dir" or (ii) if that is left undefined in the default MATLAB folder (defined by startup.m or the last folder when MATLAB was last closed). However this SPM.mat file is always loaded at the default MATLAB folder, regardless of the "spm_mat_dir" input.

1. issue: This of course returns an error that the file is not found.

Unable to read file 'SPM'. No such file or directory.

2. issue: If the analyses is run many times (as I have to do it for trial-wise beta estimations) there seem to be problems because multiple Level1Design nodes running in parallel are saving/loading the same .mat file resulting in this error:

Unable to read MAT-file /Users/jachtzehn/matlab/SPM.mat. File might be corrupt.

"/Users/jachtzehn/matlab/" is my default MATLAB startup folder.

Expected behavior

The SPM.mat file should be saved and loaded in the folder specified by "spm_mat_dir" input or at least in a node specific folder.

How to replicate the behavior

Run a level1design on any functional data included an explicit mask (defined by "mask_image" input) + change the "spm_mat_dir" folder to anything other than default.

Script/Workflow details

When "mask_image" input if defined, model.py of the spm interface adds a couple of lines to the .m file that load a SPM.mat file, modify its contents and save it again. The respective code in model.py are: line 167 - 179:

if isdefined(self.inputs.mask_image):
            # SPM doesn't handle explicit masking properly, especially
            # when you want to use the entire mask image
            postscript = "load SPM;\n"
            postscript += ("SPM.xM.VM = spm_vol('%s');\n" % simplify_list(
                self.inputs.mask_image))
            postscript += "SPM.xM.I = 0;\n"
            postscript += "SPM.xM.T = [];\n"
            postscript += ("SPM.xM.TH = ones(size(SPM.xM.TH))*(%s);\n" %
                           self.inputs.mask_threshold)
            postscript += ("SPM.xM.xs = struct('Masking', "
                           "'explicit masking only');\n")
            postscript += "save SPM SPM;\n"

As you can see, this only adds "load SPM" to the script, so during runtime the script loads the SPM.mat file from the default Matlab folder, not the level1design node folder.

Platform details:

'commit_hash': '%h',
 'commit_source': 'archive substitution',
 'networkx_version': '2.0',
 'nibabel_version': '2.2.1',
 'nipype_version': '1.1.7',
 'numpy_version': '1.13.3',
 'pkg_path': '/Users/jachtzehn/miniconda2/envs/py27_nipype/lib/python2.7/site-packages/nipype',
 'scipy_version': '1.0.0',
 'sys_executable': '/Users/jachtzehn/miniconda2/envs/py27_nipype/bin/python',
 'sys_platform': 'darwin',
 'sys_version': '2.7.14 | packaged by conda-forge | (default, Mar 30 2018, 18:21:11) \n[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]',
 'traits_version': '4.6.0'}

[jAchtzehn]

I've actually just created a workaround for this issue by changing lines 167-179 in model.py to:

if isdefined(self.inputs.mask_image):
            # SPM doesn't handle explicit masking properly, especially
            # when you want to use the entire mask image
            postscript = "load "+ self.inputs.spm_mat_dir + "/SPM.mat;\n"
            postscript += ("SPM.xM.VM = spm_vol('%s');\n" % simplify_list(
                self.inputs.mask_image))
            postscript += "SPM.xM.I = 0;\n"
            postscript += "SPM.xM.T = [];\n"
            postscript += ("SPM.xM.TH = ones(size(SPM.xM.TH))*(%s);\n" %
                           self.inputs.mask_threshold)
            postscript += ("SPM.xM.xs = struct('Masking', "
                           "'explicit masking only');\n")
            postscript += "save " + os.getcwd() + "/SPM.mat SPM;\n"

But is that OK? The SPM.mat file's size is now drastically changed...

[satra]

@jAchtzehn - are you using this within a Nipype workflow? with nodes? if so, the SPM.mat file is passed along and copied by the underlying nipype mechanisms.

[jAchtzehn]

Of course! That is the reason why I am so confused that there is a temporary SPM.mat file created in the default MATLAB folder (I have a cd command specified in my startup.m). If I delete that cd command the SPM.mat file is created in the folder I last opened with MATLAB.

[jAchtzehn]

To summarize the issue and make it a bit cleared (I hope):

When you add an explicit mask to the level1design with "mask image" input, line 167-179 are inserted in the .m file of level1design.m. These lines try to load an SPM.mat file created either (i) in the last opened MATLAB folder or (ii) in the folder specified in the startup.m file of MATLAB, I will refer to this folder as the "default MATLAB folder". It is then saved at the same position.

How this SPM.mat file is created in the first place and how this SPM.mat file makes its way back into the folder of the level1design node (or the following estimatemodel node) is not clear to me.

The issues this creates:

1. Now if we change the "spm_mat_dir" input of level1design, the SPM.mat file is created in that directory correctly, but the line "postscript = "load SPM;\n"" still means that the script will try to load an SPM.mat file that is in the default MATLAB folder, resulting in an error.

2. I have to run a trial-wise beta estimation, so for each trial I have a separate GLM. This means I run several trial analyses in parallel (up to 8). At some point, 8 different level1design nodes are running simultaneously and are trying to load/save the same SPM.mat file because it is created in the default MATLAB folder! After x amounts of analyses, this results in a corrupted SPM.mat file and all other trials afterwards crash.

[jAchtzehn]

I've done some further digging today: It seems that lines lines 167-179 in interfaces/spm/model.py are intended to load the SPM.mat inside the level1design work folder and change its contents so explicit masking is used properly by SPM (as it also says in the comments just above these lines). However, this does currently not work! After the level1design node is completed, the SPM.mat file in this folder (and also the consecutive SPM.mat file from the estimatemodel node) do not have these updated contents and explicit masking is not applied. This has caused me some trouble as I was wondering why my analyses showed up with NaN voxel (because they didn't pass the variance threshold) even though I set mask_threshold to '-Inf'!

[satra]

@jAchtzehn - i'm not sure i can follow what's going wrong here.

here is an example of how a mask file is passed on to level1design:

https://github.com/niflows/nipype1-examples/blob/master/package/niflow/nipype1/examples/fmri_spm.py#L277

could you please post your code somewhere where we can take a look? that may be better my comprehension.

[hstojic]

I'm getting errors if I set explicit mask at level 1 design as well. Code fails at the subsequent estimation step (see below). Mask seems to be specified correctly in SPM.xM, but SPM.xVol is not being set (probably in the design step?), which causes a failure in the estimation step.

190225-17:07:02,495 nipype.workflow WARNING:
	 [Node] Error on "repsup_susan6_ydiffbinsame0_l1_l1pipeline.analysis.estimate" (/media/hstojic/dataneuro/fnclearning_fmri/dProcessed/nipype_work/repsup_susan6_ydiffbinsame0_l1_l1pipeline/analysis/_subject_s057/estimate)
[Node] Error on "repsup_susan6_ydiffbinsame0_l1_l1pipeline.analysis.estimate" (/media/hstojic/dataneuro/fnclearning_fmri/dProcessed/nipype_work/repsup_susan6_ydiffbinsame0_l1_l1pipeline/analysis/_subject_s057/estimate)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-337b031fb0f4> in <module>()
      1 l1pipeline.run(
      2         'MultiProc',
----> 3         plugin_args = {'n_procs': pars['resources']['n_cores']}
      4     )

/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.pyc in run(self, plugin, plugin_args, updatehash)
    593         if str2bool(self.config['execution']['create_report']):
    594             self._write_report_info(self.base_dir, self.name, execgraph)
--> 595         runner.run(execgraph, updatehash=updatehash, config=self.config)
    596         datestr = datetime.utcnow().strftime('%Y%m%dT%H%M%S')
    597         if str2bool(self.config['execution']['write_provenance']):

/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/plugins/base.pyc in run(self, graph, config, updatehash)
    160                         if result['traceback']:
    161                             notrun.append(
--> 162                                 self._clean_queue(jobid, graph, result=result))
    163                         else:
    164                             self._task_finished_cb(jobid)

/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/plugins/base.pyc in _clean_queue(self, jobid, graph, result)
    222 
    223         if str2bool(self._config['execution']['stop_on_first_crash']):
--> 224             raise RuntimeError("".join(result['traceback']))
    225         crashfile = self._report_crash(self.procs[jobid], result=result)
    226         if jobid in self.mapnodesubids:

RuntimeError: Traceback (most recent call last):
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/spm/base.py", line 377, in _run_interface
    results = self.mlab.run()
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
    runtime = self._run_interface(runtime)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/matlab.py", line 170, in _run_interface
    self.raise_exception(runtime)
  File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 970, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
/home/hstojic/matlab/bin/matlab -nodesktop -nosplash -singleCompThread -r "addpath('/media/hstojic/dataneuro/fnclearning_fmri/dProcessed/nipype_work/repsup_susan6_ydiffbinsame0_l1_l1pipeline/analysis/_subject_s057/estimate');pyscript_estimatemodel;exit"
Standard output:

                                                              < M A T L A B (R) >
                                                    Copyright 1984-2016 The MathWorks, Inc.
                                                     R2016b (9.1.0.441655) 64-bit (glnxa64)
                                                               September 7, 2016

 
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
 
executing gpml startup script...
Executing pyscript_estimatemodel at 25-Feb-2019 17:06:43:
----------------------------------------------------------------------------------------------------
MATLAB Version: 9.1.0.441655 (R2016b)
MATLAB License Number: 649021
Operating System: Linux 4.11.0-14-generic #20~16.04.1-Ubuntu SMP Wed Aug 9 09:06:22 UTC 2017 x86_64
Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
----------------------------------------------------------------------------------------------------
MATLAB                                                Version 9.1         (R2016b)
Simulink                                              Version 8.8         (R2016b)
Bioinformatics Toolbox                                Version 4.7         (R2016b)
Communications System Toolbox                         Version 6.3         (R2016b)
Computer Vision System Toolbox                        Version 7.2         (R2016b)
Control System Toolbox                                Version 10.1        (R2016b)
Curve Fitting Toolbox                                 Version 3.5.4       (R2016b)
DSP System Toolbox                                    Version 9.3         (R2016b)
Database Toolbox                                      Version 7.0         (R2016b)
Datafeed Toolbox                                      Version 5.4         (R2016b)
Econometrics Toolbox                                  Version 3.5         (R2016b)
Financial Instruments Toolbox                         Version 2.4         (R2016b)
Financial Toolbox                                     Version 5.8         (R2016b)
Fixed-Point Designer                                  Version 5.3         (R2016b)
Fuzzy Logic Toolbox                                   Version 2.2.24      (R2016b)
Global Optimization Toolbox                           Version 3.4.1       (R2016b)
HDL Coder                                             Version 3.9         (R2016b)
Image Acquisition Toolbox                             Version 5.1         (R2016b)
Image Processing Toolbox                              Version 9.5         (R2016b)
Instrument Control Toolbox                            Version 3.10        (R2016b)
LTE System Toolbox                                    Version 2.3         (R2016b)
MATLAB Coder                                          Version 3.2         (R2016b)
MATLAB Compiler                                       Version 6.3         (R2016b)
MATLAB Compiler SDK                                   Version 6.3         (R2016b)
MATLAB Report Generator                               Version 5.1         (R2016b)
Mapping Toolbox                                       Version 4.4         (R2016b)
Model Predictive Control Toolbox                      Version 5.2.1       (R2016b)
Neural Network Toolbox                                Version 9.1         (R2016b)
Optimization Toolbox                                  Version 7.5         (R2016b)
Parallel Computing Toolbox                            Version 6.9         (R2016b)
Partial Differential Equation Toolbox                 Version 2.3         (R2016b)
Phased Array System Toolbox                           Version 3.3         (R2016b)
Robotics System Toolbox                               Version 1.3         (R2016b)
Robust Control Toolbox                                Version 6.2         (R2016b)
Signal Processing Toolbox                             Version 7.3         (R2016b)
SimBiology                                            Version 5.5         (R2016b)
Simscape                                              Version 4.1         (R2016b)
Simscape Multibody                                    Version 4.9         (R2016b)
Simscape Power Systems                                Version 6.6         (R2016b)
Simulink 3D Animation                                 Version 7.6         (R2016b)
Simulink Coder                                        Version 8.11        (R2016b)
Simulink Control Design                               Version 4.4         (R2016b)
Simulink Design Optimization                          Version 3.1         (R2016b)
Simulink Report Generator                             Version 5.1         (R2016b)
Simulink Verification and Validation                  Version 3.12        (R2016b)
Stateflow                                             Version 8.8         (R2016b)
Statistical Parametric Mapping                        Version 6906        (SPM12) 
Statistics and Machine Learning Toolbox               Version 11.0        (R2016b)
Symbolic Math Toolbox                                 Version 7.1         (R2016b)
System Identification Toolbox                         Version 9.5         (R2016b)
Trading Toolbox                                       Version 3.1         (R2016b)
Wavelet Toolbox                                       Version 4.17        (R2016b)
SPM version: SPM12 Release: 6906
SPM path: /home/hstojic/matlab2018a/toolbox/spm12/spm.m


------------------------------------------------------------------------
Running job #1
------------------------------------------------------------------------
Running 'Model estimation'

SPM12: spm_spm (v6842)                             17:06:52 - 25/02/2019
========================================================================

SPM12: spm_est_non_sphericity (v6827)              17:06:58 - 25/02/2019
========================================================================
Failed  'Model estimation'
Reference to non-existent field 'xVol'.
In file "/home/hstojic/matlab2018a/toolbox/spm12/spm_est_non_sphericity.m" (v6827), function "spm_est_non_sphericity" at line 105.
In file "/home/hstojic/matlab2018a/toolbox/spm12/spm_spm.m" (v6842), function "spm_spm" at line 431.
In file "/home/hstojic/matlab2018a/toolbox/spm12/config/spm_run_fmri_est.m" (v5809), function "spm_run_fmri_est" at line 33.

The following modules did not run:
Failed: Model estimation


Standard error:
MATLAB code threw an exception:
Job execution failed. The full log of this run can be found in MATLAB command window, starting with the lines (look for the line showing the exact #job as displayed in this error message)
------------------ 
Running job #1
------------------

File:
Name:MATLABbatch system
Line:0
Return code: 0

[hstojic]

I updated to the latest version of SPM12 (last update was in November 2018) and the errors disappear.

In fact it seems that SPM now does exactly what code in model.py lines 167 - 179 does, so this part will become unnecessary in the future.

[jAchtzehn]

I can verify that. Just manually commented out lines lines 167-179 in interfaces/spm/model.py and the SPM.mat files are identically.

@satra: please review these lines of code. They now seem to be redundant (with the latest SPM version 7487) and seem to be causing trouble on macOS systems such as mine. As I said, the extra steps coded in these lines will create a temporary SPM.mat file in the startup folder of matlab. This is not a problem unless running multiple analyses in parallel. It seems that in this case multiple MATLAB processes want to read/write the same SPM.mat file and this causes level1design to fail!

[satra]

@jAchtzehn - there seems to be two issues here:

1. the redundancy of setting the mask image in certain recent versions

if this is indeed fixed in SPM12, we can change this line:

https://github.com/nipy/nipype/blob/master/nipype/interfaces/spm/model.py#L175

to

if isdefined(self.inputs.mask_image) and '12' not in self.version.split('.')[0]

note: if this is true only of a specific revision of SPM 12, we would need to check for that more precise version.

2. reading and writing the SPM.mat file in the same location

this appears to be a function of the python script, and will be true of any script that tries to operate on the same file. this is exactly why we encourage using workflows that try to isolate directories. the other possible caveat is code that exists in "startup.m" in MATLAB. this will be executed every time matlab runs, so if this code is changing directories then you should remove those lines.

the nipype interface does make the assumption that it loads the SPM.mat file from whichever path is available to MATLAB. so if there is an SPM.mat file somewhere in MATLAB's path, that would get loaded first.

[jAchtzehn]

@jAchtzehn - there seems to be two issues here:

1. the redundancy of setting the mask image in certain recent versions

if this is indeed fixed in SPM12, we can change this line:

https://github.com/nipy/nipype/blob/master/nipype/interfaces/spm/model.py#L175

to

if isdefined(self.inputs.mask_image) and '12' not in self.version.split('.')[0]

note: if this is true only of a specific revision of SPM 12, we would need to check for that more precise version.

Agreed and should be checked with the SPM developers I suppose?

1. reading and writing the SPM.mat file in the same location

this appears to be a function of the python script, and will be true of any script that tries to operate on the same file. this is exactly why we encourage using workflows that try to isolate directories. the other possible caveat is code that exists in "startup.m" in MATLAB. this will be executed every time matlab runs, so if this code is changing directories then you should remove those lines.

I have indeed removed my code (which is just a cd ... command) from my startup.m file. The SPM.mat file then gets created at the last opened folder when MATLAB was closed. Of course, the problem of multiple instances of MATLAB trying to read/write the file persists even in that case. Please keep in mind that I have not created any special workflows, I am very close to the many examples of firstlevel analysis found in nipype's documentation. The only real difference is that I include a mask image as an input to the level1design node.

the nipype interface does make the assumption that it loads the SPM.mat file from whichever path is available to MATLAB. so if there is an SPM.mat file somewhere in MATLAB's path, that would get loaded first.

As I understand it, the .m file that is created to execute the level1design command sets the current folder to the appropriate node folder inside nipype's working directory, right? However, the extra lines added when an explicit mask is used to not seem to honour this, instead creating the temporary SPM.mat file in another location, thus creating the problem of multiple MATLAB instances accessing the same file.