mriqc docker uses .local

[marcelzwiers]

I downloaded the latest docker image (0.14.2) and converted it into a simularity image and then run into the issue that the image was working for one user, but not for another. It turned out that the cause of it was that one user had this folder ~/.local/lib/python3.6/ in his home. A possible solution could perhaps be found here?

https://stackoverflow.com/questions/38129077/how-to-run-python-without-including-local-lib-pythonx-y-site-packages-in-its

In more detail, I got the following error:

dccn-c029/0.14.2$ singularity run mriqc-0.14.2.simg -h
2019-02-28 12:31:40,194 [WARNING] DueCredit internal failure while running <function _get_inactive_due at 0x7f7f6eae42f0>: ModuleNotFoundError("No module named 'certifi'",). Please report to developers at https://github.com/duecredit/duecredit/issues (utils.py:199)
2019-02-28 12:31:40,194 duecredit.utils:WARNING DueCredit internal failure while running <function _get_inactive_due at 0x7f7f6eae42f0>: ModuleNotFoundError("No module named 'certifi'",). Please report to developers at https://github.com/duecredit/duecredit/issues
2019-02-28 12:31:40,196 [WARNING] DueCredit internal failure while running <function _get_active_due at 0x7f7f6ea71840>: ModuleNotFoundError("No module named 'certifi'",). Please report to developers at https://github.com/duecredit/duecredit/issues (utils.py:199)
2019-02-28 12:31:40,196 duecredit.utils:WARNING DueCredit internal failure while running <function _get_active_due at 0x7f7f6ea71840>: ModuleNotFoundError("No module named 'certifi'",). Please report to developers at https://github.com/duecredit/duecredit/issues
Traceback (most recent call last):
File "/usr/local/miniconda/bin/mriqc", line 11, in
load_entry_point('mriqc==0.14.2', 'console_scripts', 'mriqc')()
File "/usr/local/miniconda/lib/python3.6/site-packages/mriqc/bin/mriqc_run.py", line 162, in main
from nipype import logging as nlogging
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/init.py", line 14, in
from .refs import due
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/refs.py", line 4, in
from .external.due import due, Doi, BibTeX
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/external/due.py", line 60, in
"Imported due lacks .cite. DueCredit is now disabled")
RuntimeError: Imported due lacks .cite. DueCredit is now disabled
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/init.py", line 14, in
from .refs import due
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/refs.py", line 4, in
from .external.due import due, Doi, BibTeX
File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/external/due.py", line 60, in
"Imported due lacks .cite. DueCredit is now disabled")
RuntimeError: Imported due lacks .cite. DueCredit is now disabled

And if I rename that directory all works just fine:

dccn-c029/0.14.2$ mv ~/.local/lib/python3.6/ ~/.local/lib/python3.6.orig
dccn-c029/0.14.2$
dccn-c029/0.14.2$ singularity run mriqc-0.14.2.simg -h
usage: mriqc [-h] [--version]
[--participant_label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]]
[--session-id [SESSION_ID [SESSION_ID ...]]]
[--run-id [RUN_ID [RUN_ID ...]]]
[--task-id [TASK_ID [TASK_ID ...]]]
[-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]] [--dsname DSNAME]
[-w WORK_DIR] [--verbose-reports] [--write-graph] [--dry-run]
[--profile] [--use-plugin USE_PLUGIN] [--no-sub] [--email EMAIL]
[-v] [--webapi-url WEBAPI_URL] [--webapi-port WEBAPI_PORT]
[--upload-strict] [--n_procs N_PROCS] [--mem_gb MEM_GB]
[--testing] [-f] [--ica] [--hmc-afni] [--hmc-fsl]
[--fft-spikes-detector] [--fd_thres FD_THRES]
[--ants-nthreads ANTS_NTHREADS] [--ants-float]
[--ants-settings ANTS_SETTINGS] [--deoblique] [--despike]
[--start-idx START_IDX] [--stop-idx STOP_IDX]
[--correct-slice-timing]
bids_dir output_dir {participant,group} [{participant,group} ...]

MRIQC: MRI Quality Control

positional arguments:
bids_dir The directory with the input dataset formatted according to the BIDS standard.
output_dir The directory where the output files should be stored. If you are running group level analysis this folder should be prepopulated with the results of theparticipant level analysis.
{participant,group} Level of the analysis that will be performed. Multiple participant level analyses can be run independently (in parallel) using the same output_dir.

optional arguments:
-h, --help show this help message and exit
--version show program's version number and exit

Options for filtering the input BIDS dataset:
--participant_label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]], --participant-label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]
one or more participant identifiers (the sub- prefix can be removed)
--session-id [SESSION_ID [SESSION_ID ...]]
filter input dataset by session id
--run-id [RUN_ID [RUN_ID ...]]
filter input dataset by run id (only integer run ids are valid)
--task-id [TASK_ID [TASK_ID ...]]
filter input dataset by task id
-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]], --modalities [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]
filter input dataset by MRI type ("T1w", "T2w", or "bold")
--dsname DSNAME a dataset name

Instrumental options:
-w WORK_DIR, --work-dir WORK_DIR
change the folder to store intermediate results
--verbose-reports
--write-graph Write workflow graph.
--dry-run Do not run the workflow.
--profile hook up the resource profiler callback to nipype
--use-plugin USE_PLUGIN
nipype plugin configuration file
--no-sub Turn off submission of anonymized quality metrics to MRIQC's metrics repository.
--email EMAIL Email address to include with quality metric submission.
-v, --verbose increases log verbosity for each occurence, debug level is -vvv
--webapi-url WEBAPI_URL
IP address where the MRIQC WebAPI is listening
--webapi-port WEBAPI_PORT
port where the MRIQC WebAPI is listening
--upload-strict upload will fail if if upload is strict

Options to handle performance:
--n_procs N_PROCS, --nprocs N_PROCS, --n_cpus N_PROCS, --nprocs N_PROCS
number of threads
--mem_gb MEM_GB available total memory
--testing use testing settings for a minimal footprint
-f, --float32 Cast the input data to float32 if it's represented in higher precision (saves space and improves perfomance)

Workflow configuration:
--ica Run ICA on the raw data and include the componentsin the individual reports (slow but potentially very insightful)
--hmc-afni Use ANFI 3dvolreg for head motion correction (HMC) - default
--hmc-fsl Use FSL MCFLIRT instead of AFNI for head motion correction (HMC)
--fft-spikes-detector
Turn on FFT based spike detector (slow).
--fd_thres FD_THRES motion threshold for FD computation

Specific settings for ANTs:
--ants-nthreads ANTS_NTHREADS
number of threads that will be set in ANTs processes
--ants-float use float number precision on ANTs computations
--ants-settings ANTS_SETTINGS
path to JSON file with settings for ANTS

Specific settings for AFNI:
--deoblique Deoblique the functional scans during head motion correction preprocessing
--despike Despike the functional scans during head motion correction preprocessing
--start-idx START_IDX
Initial volume in functional timeseries that should be considered for preprocessing
--stop-idx STOP_IDX Final volume in functional timeseries that should be considered for preprocessing
--correct-slice-timing
Perform slice timing correction

[jdkent]

I'm curious if passing --cleanenv to your singularity command would also resolve this issue.

Or if specifying a new home directory when running: Documentation from singularity run --help:

-H|--home <spec>    A home directory specification.  spec can either be a
                        src path or src:dest pair.  src is the source path
                        of the home directory outside the container and dest
                        overrides the home directory within the container

cleanenv option

singularity run --cleanenv mriqc-0.14.2.simg -h

Home option

singularity run -H /some/other/directory mriqc-0.14.2.simg -h

[marcelzwiers]

I tried using --cleanenv, but that doesn't have any effect. I suppose this is because that option effects environments variables, and the issue here is the presence of a default folder on the mounted home directory (without there being an associated environment variable pointing to it).

The home option (-H) does solve the problem, but that is quite inconvenient if users have the data in their home directory...

[jdkent]

I agree, it is inconvenient to access data in the home directory, but possible:

singularity run -B /my/home/directory:/data -H /some/other/directory mriqc-0.14.2.simg -h

and in my opinion that is "less" inconvenient than having to rename a folder in you home directory back and forth depending on whether you are using singularity.

There is likely a more parsimonious solution (perhaps based on the stackoverflow question you linked), but I am having trouble thinking one up, perhaps others will have more input on this issue. I'll update if I think of anything.

[marcelzwiers]

I don't have this problem when I run a fmriprep docker/singularity-image, I don't know if that means that this issue was solved there?

[oesteban]

With docker it is really hard for your local environment to sneak into the container. However, with singularity it is really easy since $HOME is generally bound by default.

To avoid that, make sure you export PYTHONNOUSERSITE=1 and do not use the --cleanenv flag. Otherwise, if you can just delete $HOME/.local because you are not using those packages anymore that should also do.

I'm adding setting PYTHONNOUSERSITE for the next release of MRIQC.

[marcelzwiers]

With docker it is really hard for your local environment to sneak into the container. However, with singularity it is really easy since $HOME is generally bound by default.

To avoid that, make sure you export PYTHONNOUSERSITE=1 and do not use the --cleanenv flag. Otherwise, if you can just delete $HOME/.local because you are not using those packages anymore that should also do.

I'm adding setting PYTHONNOUSERSITE for the next release of MRIQC.

Thanks, that worked!