Singularity image trying to access Amazon S3

[dmd]

In 1.5.0, the Singularity image fails with:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2aaacc203198>: Failed to establish a new connection: [Errno 110] Connection timed out'))

[dmd]

This fails because compute nodes, at our institution, do not have internet access. Assuming this is something 1.5.0 needs (this didn't happen in 1.4.1), is there a way to tell the process doing this inside the container to go via a proxy?

[effigies]

I believe this is and #1777 are duplicates of #1699, but I might be wrong...

Can you try:

export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

[dmd]

I'm having trouble understand what's wanted here. What of the following do I need?

1. Does the container need internet access, which case I must set SINGULARITYENV_https_proxy?

2. Do I need to map /home/fmriprep (-B $HOME:/home/fmriprep) ?

3. Do I need SINGULARITYENV_TEMPLATEFLOW_HOME?

Or some combination of these?

[oesteban]

1. Does the container need internet access, which case I must set SINGULARITYENV_http_proxy?

Yes, it does. However, you can pre-download everything you need and run on a node without internet access.

2. Do I need to map /home/fmriprep (-B $HOME:/home/fmriprep) ?

Although this might do, I would recommend to map only the templateflow folder (see below)

3. Do I need SINGULARITYENV_TEMPLATEFLOW_HOME?

I think this is the easiest route to combine the answers to your three questions.

Binding a TemplateFlow home

This would be the recommendation assuming your nodes have internet access:

1. Prepare a folder where templates will be stored (i.e., a persistent file-system, which you probably want to make available to a group of users that will run fMRIPrep). Let's say this path in the host is /share/group/TemplateFlow.
2. Before running the singularity image, tell Singularity that you want to set up some environment variables. In particular, you'll change the templateflow home folder:

export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow

3. When running fMRIPrep, bind the templateflow folder: -B /share/group/TemplateFlow:/home/fmriprep/.cache/templateflow.

Notes:

• Some singularity installations auto-bind $HOME. Of those, some will update the $HOME variable and some won't. This is very problematic, and that's why I'm suggesting to bind only the templateflow folder AND setting $SINGULARITYENV_TEMPLATEFLOW_HOME.
• Some singularity installations will not allow you to bind a directory into a path that does not exist in the image. This is why I'm suggesting /home/fmriprep/.cache/fmriprep for the bind point.
• Make sure that fMRIPRep users have permissions to access /share/group/TemplateFlow.
• Make sure you run singularity with the flag --cleanenv so that the local $TEMPLATEFLOW_HOME does not leak into the container.

When you don't have internet access from compute nodes.

You'll need to prepare the /share/group/TemplateFlow accordingly. The easiest route is using a Python 3 environment.

1. Point templateflow to the shared folder:

export TEMPLATEFLOW_HOME=/share/group/TemplateFlow

2. Install templateflow

python -m pip install templateflow

3. Fetch those templates you plan to use:

python -c "from templateflow import api; api.get('MNI152NLin2009cAsym')"
python -c "from templateflow import api; api.get('OASIS30ANTs')"

4. Follow the previous guidelines (when internet access is available), and make sure you run fMRIPrep with the flag --notrack to avoid further internet access attempts.

[dmd]

Thanks for this.

If I only bind:

-B /data/TemplateFlow:/home/fmriprep/.cache/templateflow

then I get:

Traceback (most recent call last): File "/usr/local/miniconda/bin/fmriprep", line 10, in <module> sys.exit(main()) File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 311, in main opts = get_parser().parse_args() File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 284, in get_parser latest = check_latest() File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/version.py", line 22, in check_latest cachefile.parent.mkdir(parents=True, exist_ok=True) File "/usr/local/miniconda/lib/python3.7/pathlib.py", line 1241, in mkdir self._accessor.mkdir(self, mode) OSError: [Errno 30] Read-only file system: '/home/fmriprep/.cache/fmriprep'

It looks like it still wants to write there.

My complete command now is:

export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow
/cm/shared/singularity/bin/singularity run \
    -B /data:/data -B /data1:/data1 -B /data2:/data2 -B /data3:/data3 -B /cm/shared:/cm/shared \
    -B /data/TemplateFlow:/home/fmriprep/.cache/templateflow \
    --cleanenv \
    /cm/shared/singularity/images/fmriprep-1.5.0.simg \
    /data/ajanes/testing \
    /data/ajanes/testing/derivatives \
    participant \
    --fs-license-file /cm/shared/freesurfer-6.0.1/license.txt \
    --participant_label acj \
    --output-spaces "MNI152NLin2009cAsym:res-2 anat func fsaverage" \
    --n_cpus 4 \
    --mem-mb 8192 \
    --notrack \
    --use-plugin /data/ddrucker/workaround.yml \
    --use-aroma --ignore-aroma-denoising-errors \
    --use-syn-sdc \
    -w  /data/ajanes/testing/fmriprep-work

[oesteban]

Not sure if this will make any difference, but please note the folder it is complaining about /home/fmriprep/.cache/fmriprep (instead of /home/fmriprep/.cache/templateflow)

There must be a glitch somewhere in your command line.

[dmd]

My command line is exactly as pasted above...

[effigies]

Oh, this is the version check that's causing the problem in #1777, @oesteban.

[oesteban]

Can you ls /data/TemplateFlow ?

[dmd]

$ ls -l /data/TemplateFlow/
total 64
drwxrwxr-x  2 ddrucker ddrucker  4096 Sep 16 13:54 tpl-fsaverage
drwxrwxr-x  3 ddrucker ddrucker  4096 Sep 16 13:54 tpl-fsLR
drwxrwxr-x  3 ddrucker ddrucker  4096 Sep 16 13:54 tpl-MNI152Lin
drwxrwxr-x  2 ddrucker ddrucker 12288 Sep 16 13:54 tpl-MNI152NLin2009cAsym
drwxrwxr-x  3 ddrucker ddrucker  8192 Sep 16 13:54 tpl-MNI152NLin6Asym
drwxrwxr-x  3 ddrucker ddrucker  4096 Sep 16 13:54 tpl-MNI152NLin6Sym
drwxrwxr-x 14 ddrucker ddrucker  4096 Sep 16 13:54 tpl-MNIInfant
drwxrwxr-x  9 ddrucker ddrucker  4096 Sep 16 13:54 tpl-MNIPediatricAsym
drwxrwxr-x  3 ddrucker ddrucker  4096 Sep 16 13:54 tpl-NKI
drwxrwxr-x  2 ddrucker ddrucker  4096 Sep 16 13:54 tpl-OASIS30ANTs
drwxrwxr-x  2 ddrucker ddrucker  4096 Sep 16 13:54 tpl-PNC

[dmd]

Does that need to be writable by other users, or just readable?

[oesteban]

Judging by #1778 (comment) I'm now inclined to think you just found a nasty bug.

EDIT: if all templates are downloaded, it works if it is just readable (and folders executable)

[dmd]

It's definitely trying to write to /home/fmriprep/.cache/fmriprep

[dmd]

I'm getting on STDERR:

Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_desc-brain_mask.nii.gz

and then

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin6Asym/tpl-MNI152NLin6Asym_res-01_desc-brain_mask.nii.gz (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x2aaadb08d748>: Failed to establish a new connection: [Errno 110] Connection timed out'))

Even though I have
export SINGULARITYENV_TEMPLATEFLOW_HOME=/data/TemplateFlow
and
-B /data/TemplateFlow:/data/TemplateFlow

and in #1778 (comment) you can see I have that downloaded...

[dmd]

If I give it a https_proxy, it succeeds in downloading, so it doesn't fail ... even though it then doesn't actually write it anywhere.

My TEMPLATEFLOW_HOME is read-only, so it didn't write it there, and it didn't write it to .cache/fmriprep either (the only file it wrote there is latest).

It looks to me like you're attempting to download the file even if you already have it in TEMPLATEFLOW_HOME, and then just ignoring the result?