Provide better documentation for SPM standalone setup #820

ghisvail · 2022-11-29T17:36:30Z

Discussed in #819

^{Originally posted by AudreyDuran November 28, 2022}
Hi,

I am trying to run t1-volume pipeline with SPM12 standalone and clinica v0.7.2, on a custom dataset.

Here are some environment variables values defined in my ~/.bashrc:

SPM_HOME="/opt/spm12"
SPM_VERSION=12
SPM_REVISION=r7771
MATLAB_VERSION=R2019b
MCR_VERSION=v97
MCR_UPDATE=9
MCR_INHIBIT_CTF_LOCK=1
MATLABCMD="${SPM_HOME}/run_spm12.sh /opt/mcr/${MCR_VERSION} script"

When I run the pipeline, I get the following output:

2022-11-28 10:17:48,336:INFO:The t1-volume pipeline is divided into 4 parts:
        t1-volume-tissue-segmentation pipeline: Tissue segmentation, bias correction and spatial normalization to MNI space
        t1-volume-create-dartel pipeline: Inter-subject registration with the creation of a new DARTEL template
        t1-volume-dartel2mni pipeline: DARTEL template to MNI
        t1-volume-parcellation pipeline: Atlas statistics
2022-11-28 10:17:55,639:INFO:Part 1/4: Running t1-volume-segmentation pipeline.




2022-11-28 10:25:57,830:INFO:The pipeline will be run on the following 1282 image(s):
2022-11-28 10:25:57,830:INFO:   sub-00098E7F | ses-w96, ses-w0,
2022-11-28 10:25:57,830:INFO:   sub-00331D3D | ses-w0, ses-w96,
2022-11-28 10:25:57,830:INFO:   sub-003D930B | ses-w0, ses-w96,
2022-11-28 10:25:57,830:INFO:   sub-00438273 | ses-w96, ses-w0,
2022-11-28 10:25:57,830:INFO:   sub-006ACF86 | ses-w96, ses-w0,
2022-11-28 10:25:57,830:INFO:   sub-0095A525 | ses-w0, ses-w96,
2022-11-28 10:25:57,830:INFO:   sub-00DBEADC | ses-w0, ses-w96,
2022-11-28 10:25:57,830:INFO:   sub-01BA0A03 | ses-w96, ses-w0,
2022-11-28 10:25:57,830:INFO:   sub-01D4AE69 | ses-w0, ses-w96,
2022-11-28 10:25:57,830:INFO:   sub-0271F164 | ses-w0, ses-w96,
2022-11-28 10:25:57,830:INFO:   sub-02D7D496 | ses-w96, ses-w0,
2022-11-28 10:25:57,831:INFO:   sub-0391403C | ses-w96, ses-w0,
2022-11-28 10:25:57,831:INFO:   sub-03BCF734 | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-04550C9F | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-047D2FA8 | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-04CB17EC | ses-w96, ses-w0,
2022-11-28 10:25:57,831:INFO:   sub-05A8B309 | ses-w96, ses-w0,
2022-11-28 10:25:57,831:INFO:   sub-06152DED | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-0693D50B | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-069B4AC2 | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-07AB59AD | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-07E02FCB | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-07E24E6C | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   sub-0844A958 | ses-w96, ses-w0,
2022-11-28 10:25:57,831:INFO:   sub-08BB1347 | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:   ...
2022-11-28 10:25:57,831:INFO:   sub-FFCE393E | ses-w0, ses-w96,
2022-11-28 10:25:57,831:INFO:The pipeline will last approximately 10 minutes per image.
Traceback (most recent call last):
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/utils/spm_docs.py", line 49, in _strip_header
    index = doc.index(hdr)
ValueError: substring not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/thera/clinicaEnv/bin/clinica", line 8, in <module>
    sys.exit(main())
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/cmdline.py", line 89, in main
    cli()
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/t1_volume/t1_volume_cli.py", line 91, in cli
    ctx.invoke(
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/t1_volume_tissue_segmentation/t1_volume_tissue_segmentation_cli.py", line 68, in cli
    else pipeline.run()
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/engine.py", line 261, in run
    self.build()
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/engine.py", line 25, in func_wrapper
    res = func(self, *args, **kwargs)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/engine.py", line 232, in build
    self.build_core_nodes()
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/t1_volume_tissue_segmentation/t1_volume_tissue_segmentation_pipeline.py", line 163, in build_core_nodes
    new_segment = npe.Node(spm.NewSegment(), name="2-SpmSegmentation")
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/preprocess.py", line 1873, in __init__
    _local_version = SPMCommand().version
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/base.py", line 310, in __init__
    super(SPMCommand, self).__init__(**inputs)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 197, in __init__
    unavailable_traits = self._check_version_requirements(
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 295, in _check_version_requirements
    if names and self.version:
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/base.py", line 358, in version
    info_dict = Info.getinfo(
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/base.py", line 239, in getinfo
    out = sd._strip_header(out.runtime.stdout)
  File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/utils/spm_docs.py", line 51, in _strip_header
    raise IOError("This docstring was not generated by Nipype!\n") from e
OSError: This docstring was not generated by Nipype!

Do you have any idea about what the problem is ?

I also get the same ValueError when running

from nipype.interfaces import spm
matlab_cmd = ''/opt/spm12/run_spm12.sh /opt/mcr/v97 script''
spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True)
spm.SPMCommand().version

You can find bellow some platform details:

{‘nibabel_version’: ‘2.5.2’,
‘nipype_version’: ‘1.8.5’,
‘numpy_version’: ‘1.23.3’,
‘pkg_path’: ‘/clinicaEnv/lib/python3.8/site-packages/nipype’,
‘scipy_version’: ‘1.9.2’,
‘sys_executable’: ‘/clinicaEnv/bin/python’,
‘sys_platform’: ‘linux’,
‘sys_version’: ‘3.8.5 (default, Jan 27 2021, 15:41:15) \n[GCC 9.3.0]’,
‘traits_version’: ‘6.3.2’}

Thanks a lot for your help !!

The text was updated successfully, but these errors were encountered:

ghisvail · 2022-11-29T17:49:46Z

The solution was to set FORCE_SPMMCR which is used as a fallback to setting use_mcr to True.

This raises a wider discussion regarding our support for both MATLAB and standalone distributions of SPM. Support for both can either be controlled explicitly with the use_mcr and matlab_cmd arguments, or implicitly with the FORCE_SPMMCR and MATLABCMD envvars. See Nipype's documentation.

Both have their pros and cons as always.

NicolasGensollen · 2022-11-30T08:19:26Z

I haven't looked into it, so apologies if this is a stupid question, but isn't this function supposed to handle the SPM standalone config ?

clinica/clinica/utils/spm.py

Lines 101 to 129 in 3a4981b

    
           def use_spm_standalone(): 
        
               """Use SPM Standalone with MATLAB Common Runtime.""" 
        
               import os 
        
               import platform 
        
               from nipype.interfaces import spm 
        
               from clinica.utils.stream import cprint 
        
               # This section of code determines whether to use SPM standalone or not 
        
               if all(elem in os.environ.keys() for elem in ["SPMSTANDALONE_HOME", "MCR_HOME"]): 
        
                   spm_standalone_home = os.getenv("SPMSTANDALONE_HOME") 
        
                   mcr_home = os.getenv("MCR_HOME") 
        
                   if os.path.exists(spm_standalone_home) and os.path.exists(mcr_home): 
        
                       cprint("SPM standalone has been found and will be used in this pipeline") 
        
                       if platform.system().lower().startswith("darwin"): 
        
                           matlab_cmd = ( 
        
                               f"cd {spm_standalone_home} && ./run_spm12.sh {mcr_home} script" 
        
                           ) 
        
                       elif platform.system().lower().startswith("linux"): 
        
                           matlab_cmd = f"{os.path.join(spm_standalone_home, 'run_spm12.sh')} {mcr_home} script" 
        
                       else: 
        
                           raise SystemError("Clinica only support macOS and Linux") 
        
                       spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True) 
        
                       cprint(f"Using SPM standalone version {spm.SPMCommand().version}") 
        
                   else: 
        
                       raise FileNotFoundError( 
        
                           "$SPMSTANDALONE_HOME and $MCR_HOME are defined, but linked to non existent folder " 
        
                       )

ghisvail · 2022-11-30T09:31:27Z

so apologies if this is a stupid question, but isn't this function supposed to handle the SPM standalone config ?

The question is absolutely valid. Indeed this function is supposed to handle the SPM config. But there are multiple flaws I think:

It is always paired with another utility function called spm_standalone_is_available within which there is some duplicated logic wrt the detection of the envvars. A path to improvement would be to refactor them both into a use_spm_standalone_if_available function.
As stated in the Nipype's documentation: "In order to use the standalone MCR version of spm, you need to ensure that the following commands are executed at the beginning of your script". Which makes sense since the proposed solution monkey-patch (yikes) the base SPMCommand from which all SPM interfaces inherit from. But due to the complicated logic in our pipelines, it's not impossible that some SPM interface has already been imported before calling use_spm_standalone.
The answered solution works because Nipype's envvars take precedence over base class setup.

ghisvail · 2022-11-30T09:41:34Z

In my opinion, instead of depending on monkey-patching, we should just let Nipype logic handle it:

matlab_cmd (str): Sets the default matlab command. If None, the value of the environment variable SPMMCRCMD will be used if set and use_mcr is True or the environment variable FORCE_SPMMCR is set. If one of FORCE_SPMMCR or SPMMCRCMD is not set, the existence of the environment variable MATLABCMD is checked and its value is used as the matlab command if possible. If none of the above was successful, the fallback value of ‘matlab -nodesktop -nosplash’ will be used.

i.e.:

To support SPM standalone only, set SPMMCRCMD and FORCE_SPMMCR
To support SPM MATLAB only, set MATLABCMD and do not set FORCE_SPMMCR
To support both with a fallback or a switch, set MATLABCMD and set or unset FORCE_SPMMCR

NicolasGensollen · 2022-12-02T13:45:20Z

clinica/clinica/utils/spm.py

Lines 80 to 98 in 444e4a4

    
           def spm_standalone_is_available(): 
        
               """Tell if SPM standalone can be used. 
        
               Returns: 
        
                   True if SPM standalone is detected, False otherwise. Note that it does not guarantee that SPM (classical) is 
        
                   up and running in the system. 
        
               """ 
        
               import os 
        
               from os.path import expandvars, isdir 
        
               use_spm_stand = False 
        
               if all(elem in os.environ.keys() for elem in ["SPMSTANDALONE_HOME", "MCR_HOME"]): 
        
                   if isdir(expandvars("$SPMSTANDALONE_HOME")) and isdir(expandvars("$MCR_HOME")): 
        
                       use_spm_stand = True 
        
                   else: 
        
                       raise FileNotFoundError( 
        
                           "[Error] $SPMSTANDALONE_HOME and $MCR_HOME are defined, but linked to non existent folder" 
        
                       ) 
        
               return use_spm_stand

github-actions · 2023-03-03T04:06:47Z