Running mriqc without internet connection

[asoroosh]

Hi folks, great great app!

tl;dr: how can I run mriqc without internet connection?

Just a bit of background; I am running mriqc as a singularity image on a cluster which is isolated from the internet. So, if I run the following call

singularity run \ 
-B /data/unprocessed,/data/processed \ 
/apps/software/containers/mriqc-0.15.2rc1.sif \   
${InputDir} \
${OutputDir} \
-w ${OutputDir} \ 
participant --participant-label ${SubID} \  
--modalities ${ImgTyp} \                                                                                                      
--no-sub

I get the following error.

Downloading https://templateflow.s3.amazonaws.com/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 594, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 815, in _prepare_proxy
    conn.connect()
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connection.py", line 324, in connect
    self._tunnel()
  File "/usr/local/miniconda/lib/python3.7/http/client.py", line 911, in _tunnel
    message.strip()))
OSError: Tunnel connection failed: 403 Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/miniconda/lib/python3.7/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 418, in init_mriqc
    wf_list.append(build_workflow(dataset[mod], mod, settings=settings))
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/core.py", line 27, in build_workflow
    workflow = anat_qc_workflow(dataset, mod=mod, settings=settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 87, in anat_qc_workflow
    norm = spatial_normalization(settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 191, in spatial_normalization
    get_template(tpl_id, resolution=resolution, desc='brain', suffix='mask'))
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 39, in get
    _s3_get(filepath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 130, in _s3_get
    r = requests.get(url, stream=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/requests/adapters.py", line 507, in send
    raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='templateflow.s3.amazonaws.com', port=443): Max retries exceeded with url: /tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))

I then (naively?) clone https://github.com/templateflow/templateflow into my $HOME and add the following before my singularity call:

export SINGULARITYENV_TEMPLATEFLOW_HOME=${HOME}/templateflow

But then I bump into another error:

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 418, in init_mriqc
    wf_list.append(build_workflow(dataset[mod], mod, settings=settings))
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/core.py", line 27, in build_workflow
    workflow = anat_qc_workflow(dataset, mod=mod, settings=settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 87, in anat_qc_workflow
    norm = spatial_normalization(settings)
  File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/workflows/anatomical.py", line 191, in spatial_normalization
    get_template(tpl_id, resolution=resolution, desc='brain', suffix='mask'))
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'reference_mask' of a RobustMNINormalizationInputSpecRPT instance is an existing file name, but the path  '[]' does not exist.

Is setting SINGULARITYENV_TEMPLATEFLOW_HOME to a locally cloned templatefelow sensible? If so, what about RobustMNINormalizationInputSpecRPT instance is an existing file name, but the path '[]' does not exist. error?

Thank you so much in advance!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[oesteban]

Sorry @asoroosh. For tips on running our singularity containers without internet access, please check this out and this.

[oesteban]

Updated documentation: https://www.nipreps.org/apps/singularity/#templateflow-and-singularity