1.4.1 templateflow proxy error #1699
Comments
Correct. Your firewall is preventing templates from being pulled down. |
|
I am having the same problem but with
|
|
And I assume you checked you can actually write to ? |
|
I am using singularity image |
|
Okay, then that is potentially a different issue. What is fMRIPrep version? |
|
the current version now, 1.4.1 |
|
And your full command line? |
|
I was trying to run with different output spaces
|
|
@a3sha2 I believe this -> https://neurostars.org/t/antsbrainextraction-failure-freesurfer-does-fine/3949/27?u=oesteban would solve your issues. |
|
@oricon I believe this -> https://neurostars.org/t/problems-using-pediatric-template-from-templateflow/4566/15?u=oesteban would solve your issues. |
|
I am getting the same error as @a3sha2. After reading the possible solutions (https://neurostars.org/t/antsbrainextraction-failure-freesurfer-does-fine/3949/32 -> export SINGULARITYENV_TEMPLATEFLOW_HOME=[..]) it is still unclear to me how to use this approach when running multiple singularity instances in parallel. Would that not give me concurrency problems when they al start downloading the same template? Do I have to prepare all the templates in advance? |
|
This is what I see (e.g. many templates size zero): |
|
I downloaded the templates I was using to a templateflow directory and this
worked fine with multiple instances.
Joseph M. Orr, PhD
Assistant Professor
Department of Psychological and Brain Sciences
Texas A&M Institute for Neuroscience
Texas A&M University
College Station, TX
…On Tue, Sep 3, 2019, 3:58 AM Marcel Zwiers ***@***.***> wrote:
This is what I see:
singularity shell /opt/fmriprep/1.4.1/fmriprep-1.4.1.simg
Singularity: Invoking an interactive shell within container...
Singularity fmriprep-1.4.1.simg:/home/mrphys/marzwi> echo $HOME
/home/fmriprep
Singularity fmriprep-1.4.1.simg:/home/mrphys/marzwi> ls -al $HOME/.cache/templateflow/tpl-OASIS30ANTs/
total 57544
drwxrwxrwx 2 root root 972 Aug 5 13:56 .
drwxrwxrwx 13 root root 295 Jul 9 13:37 ..
-rw-rw-rw- 1 root root 39 Jun 8 12:48 CHANGES
-rw-rw-rw- 1 root root 744 Jun 8 12:48 template_description.json
-rw-rw-rw- 1 root root 537 Jun 8 12:48 template_sample.tsv
-rw-rw-rw- 1 root root 32363109 Jun 8 12:48 tpl-OASIS30ANTs_res-01_T1w.nii.gz
-rw-rw-rw- 1 root root 292399 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-4_dseg.nii.gz
-rw-rw-rw- 1 root root 164 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-4_dseg.tsv
-rw-rw-rw- 1 root root 303542 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-6_dseg.nii.gz
-rw-rw-rw- 1 root root 205 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-6_dseg.tsv
-rw-rw-rw- 1 root root 264866 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-BrainCerebellumExtraction_mask.nii.gz
-rw-rw-rw- 1 root root 255798 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
-rw-rw-rw- 1 root root 5082326 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-brain_T1w.nii.gz
-rw-rw-rw- 1 root root 182192 Jun 8 12:48 tpl-OASIS30ANTs_res-01_desc-brain_mask.nii.gz
-rw-rw-rw- 1 root root 446641 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-BS_probseg.nii.gz
-rw-rw-rw- 1 root root 887554 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-CBM_probseg.nii.gz
-rw-rw-rw- 1 root root 5260423 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-CGM_probseg.nii.gz
-rw-rw-rw- 1 root root 6205417 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-CSF_probseg.nii.gz
-rw-rw-rw- 1 root root 726641 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-SCGM_probseg.nii.gz
-rw-rw-rw- 1 root root 4057922 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-WM_probseg.nii.gz
-rw-rw-rw- 1 root root 2589048 Jun 8 12:48 tpl-OASIS30ANTs_res-01_label-brain_probseg.nii.gz
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_poldracklab_fmriprep_issues_1699-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DACBNSL6TNPE7XGGEAJAEJB3QHYRKPA5CNFSM4IBVK6F2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5XQOEA-23issuecomment-2D527370000&d=DwMCaQ&c=u6LDEWzohnDQ01ySGnxMzg&r=r5Mpc3mGXfHYvMphItda23EYv_4RBAqWoRqiqZo1OLo&m=zw4WZUSaSyWcIaGhEtchdqctNPNtAJr9Zn8gndwOMa0&s=wgISNKlXc3m3M-2STa28jGyde9HQVRiKze8Weax3r_w&e=>,
or mute the thread
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ACBNSL4FTSTTTYDA2ACW3YDQHYRKPANCNFSM4IBVK6FQ&d=DwMCaQ&c=u6LDEWzohnDQ01ySGnxMzg&r=r5Mpc3mGXfHYvMphItda23EYv_4RBAqWoRqiqZo1OLo&m=zw4WZUSaSyWcIaGhEtchdqctNPNtAJr9Zn8gndwOMa0&s=2TbrICwfUhbFiFaVuIoAZgEUvI26BikYmJXxrotn4Ok&e=>
.
|
|
@marcelzwiers
fmriprep1.4.1.simg python -c “from templateflow import api; print(api.get(‘MNI152Lin’,‘MNI152NLin2009cAsym’,‘MNI152NLin6Asym’,‘MNI152NLin6Sym’,‘PNC’,resolution=2))” ```
|
|
then
|
|
@marcelzwiers did @a3sha2's suggestions work for you? |
|
Maybe it could work for me, but it is a major pain for our center. I maintain a fmriprep module on our central storage that hundreds of researchers have access to. I cannot tell all of them to follow a3sha2 suggestion for every fmriprep run they want to do (people here will just stick to using the old version). Is there a plan to solve this issue for future fmriprep versions? |
|
it will work, we also use it like that on our cluster at Penn @marcelzwiers |
|
It should also be possible to set up a single cache for the entire cluster and include the environment variable in your module so that the templates only need to be downloaded once, and shared. @oesteban correct me if I'm wrong, but as long as you're using a single version of fMRIPrep, there should be no chance of needing to fetch new templates, so you could have a many-to-one relationship of fMRIPrep versions to templateflow caches. |
|
@effigies It is a bit confusing to me. So what I understood is that there is an incomplete .cache in the 1.4.1 singularity container (it is unclear to me why all the standard templates are not just simply put there), and I should manually create a single read-only(?) template cache folder on central storage that serves the different fmriprep versions on our cluster. The alternative is that every user should do what @a3sha2 does (i.e. first run the api.get command to get the templates they need) in their own home-directory and I make the settings correct for them (i.e. do the binding) in the frmiprep common.tcl module file. Both options seem like a major step down for us, compared to the previous fmriprep versions |
|
TBH it's not really clear to me why these aren't showing up on Singularity images. They should be present in the Docker images, and Singularity should replicate that. But I believe Oscar has looked into this more. Possibly we're downloading the templates into a directory that doesn't get searched, when Singularity pulls in most of the host environment? |
|
Ah, yes: We're pulling them into And in future versions, we can set |
|
Thanks @effigies I will try that first |
|
NB, there are many standard templates of size zero in the singularity container, is that different in the docker? See e.g.: |
|
That's the same in the Docker. The Docker image only provides the three we use by default, but the templateflow package is intended to aggregate a lot of of templates, not just as a component of fMRIPrep. So you will run into issues if people want to use templates besides these, at which point being able to provide alternative locations for the cache will be useful. |
|
Maybe it would be good to add that to the docs, that these are the fmriprep standard templates... It now says: --output-spaces Standard and non-standard spaces to resample anatomical and functional images to. Standard spaces may be specified by the form [:res-][:cohort-][...], where is a keyword (valid keywords: “MNI152Lin”, “MNI152NLin2009cAsym”, “MNI152NLin6Asym”, “MNI152NLin6Sym”, “MNIInfant”, “MNIPediatricAsym”, “NKI”, “OASIS30ANTs”, “PNC”, “fsLR”, “fsaverage”) or path pointing to a user-supplied template, and may be followed by optional, colon-separated parameters. Non-standard spaces (valid keywords: anat, T1w, run, func, sbref, fsnative) imply specific orientations and sampling grids. Important to note, the res-* modifier does not define the resolution used for the spatial normalization |
|
Some notes:
Correct. The only caveat is that the folder bound must exist prior to binding within the singularity image. Perhaps we should add a
If you play with
I agree with this, but on the other end of the rope there are people willing to work with pediatric templates or nonhuman templates who badly needed the flexibility. We are completely open to implementing any ideas that make both types of users happy. |
Can't the container be made such they they then download these templates to a scratch space inside the container? Or to their home, working-directory or any user-specified directory? |
This is the default for Docker, however you need elevated privileges to run Singularity with write permissions. |
|
I personally don't believe there is much added value to using non-standard templates, especially given the brilliant job ANTs is doing. So we were just trying to use a standard template, but ran into these issues... |
As I said, we are very happy to work towards covering all use-cases and at the same time minimize the friction. |
|
Why not use the working-directory as a scratch space instead of /home/fmriprep/.cache/templateflow? |
|
How would you know that path to pre-fetch templates? Conversely, nothing stops you from setting: We need to consider here something that works well for Docker users too. Maybe, if we take steps towards building Singularity images separately from Docker images then we will have space for these tailored tweaks. |
I clearly don't understand the problem-space you are dealing with because I don't understand your answer :-). Anyhow, thanks very much for your help, we are already very happy with default templates that work (which I'm testing right now) |
|
p.s. with working directory I was referring to [-w WORK_DIR], not the current/parent working directory |
As you correctly assessed, the image is built with only a subset of TemplateFlow objects downloaded. The rest of them are just a stub of zero-sized files that are dynamically downloaded and replaced the first time you try to use them via templateflow. Therefore, I'm referring to the problem of downloading info to a path that is unknown at image build time. For your use case, if you are to rewrite the default before running singularity. The first time you do this, it would pull down the templates required by your |
|
export SINGULARITYENV_TEMPLATEFLOW_HOME=/home/fmriprep/.cache/templateflow works well for the standard templates. I thought about it a little more, to allow for non-standard templates, and decided to copy the templateflow directory from the container to a directory that is writable for all users and set that as the SINGULARITYENV_TEMPLATEFLOW_HOME. There may be some concurrency problems (two users writing the same template) or even security problems (as any of our users can put any file there), but I think they will be minor or never happen in the real world. Thanks again for your help, I think fmriprep is a great piece of software! |
|
Hi @marcelzwiers, could you check whether the Further comments and explanations can be found in #1778 (comment). |
|
I think this has been addressed. Please reopen if necessary. |
|
Hi all, while it seems this problem has a solution in singularity via SINGULARITY_TEMPLATEFLOW_HOME, I have not been able to find a similar workaround using the FMRIPREP Docker container. It seems that $HOME is also reset there, and I tried passing in $TEMPLATEFLOW_HOME as an -e flag to a user-independent location on our server, without any luck. Once FMRIPREP is running, it is still looking for templates in /home/fmriprep/.cache/templateflow. Any help would be greatly appreciated! @oesteban Here's my current script: |
I'm getting a proxy error with a 1.4.1 singularity build. It looks like OASIS ANTs template is being downloaded when fmriprep starts and its being blocked. Is there a workaround for this where the templates don't need to download? Or would I need to have the sysadmin unblock https://templateflow.s3.amazonaws.com from firewalls? I'm not sure if its a firewall issue or just a policy of not allowing tunnels from compute nodes. I also have Sentry errors with reports being blocked, but the --notrack option eliminates these errors. The error log is pasted below.
Thanks
The text was updated successfully, but these errors were encountered: