Skip to content

Parallel execution of nipype failing #1127

@pintohutch

Description

@pintohutch

Hello,

When submitting a nipype workflow execution script in parallel with a job scheduler (in this case we're using SGE), nipype crashes when more than one job tries to create the ~/.nipype directory (as one of the jobs creates it before the others). Specifically:

File "/usr/local/lib/python2.7/dist-packages/CPAC/anat_preproc/anat_preproc.py", line 1, in
from nipype.interfaces.afni import preprocess
File "/usr/local/lib/python2.7/dist-packages/nipype/init.py", line 10, in
config = NipypeConfig()
File "/usr/local/lib/python2.7/dist-packages/nipype/utils/config.py", line 64, in init
os.makedirs(config_dir)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/root/.nipype'

I think what is happening is that multiple jobs actually make it through the if statement on https://github.com/nipy/nipype/blob/master/nipype/utils/config.py#L65, then one of the jobs creates the directory before the others. Then when the other jobs attempt to create that directory, they get the OSError exception.

This can be fixed by adding a try block around the https://github.com/nipy/nipype/blob/master/nipype/utils/config.py#L66 that handles if it raises an OSError in the case that folder exists.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions