Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TemplateFlow permissions error in 1.3.0 #1500

Closed
effigies opened this issue Feb 8, 2019 · 23 comments

Comments

@effigies
Copy link
Collaborator

commented Feb 8, 2019

Relevant traceback:

  File "/usr/local/miniconda/lib/python3.7/site-packages/smriprep/workflows/anatomical.py", line 621, in init_skullstrip_ants_wf
    t1_skull_strip.inputs.brain_template = get_template(skull_strip_template, 'res-01_T1w.nii.gz')
  File "/usr/local/miniconda/lib/python3.7/site-packages/templateflow/api.py", line 24, in get
    out_file = api.get(filepath)
[...]
  File "/usr/local/miniconda/lib/python3.7/site-packages/datalad/cmd.py", line 530, in run
    raise CommandError(str(cmd), msg, status, out[0], out[1])
datalad.support.exceptions.CommandError: CommandError: command '['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'find', '--json', '--not', '--in', 'here', '--', 'tpl-OASIS30ANTs_res-01_T1w.nii.gz']' failed with exitcode 1
Failed to run ['git', '-c', 'receive.autogc=0', '-c', 'gc.auto=0', 'annex', 'find', '--json', '--not', '--in', 'here', '--', 'tpl-OASIS30ANTs_res-01_T1w.nii.gz'] under '/opt/templateflow/tpl-OASIS30ANTs'. Exit code=1. out= err=git-annex: .git/annex/journal.lck: openFd: permission denied (Permission denied)

My suspicion is that the solution would be to add umask 000 before datalad commands in the Dockerfile.

Full log.

cc @Gilles86 @oesteban

@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

I'm still getting the error with a singularity image built off the new docker image:

Singularity poldracklab_fmriprep_1.3.0-1-2019-02-08-1b5ad521ee2a.simg:/scratch/users/oesteban/fmriprep-ds000003> ls -lah /opt/templateflow
total 1.5K
drwxrwxrwx 12 root root  330 Feb  8 12:23 .
drwxr-xr-x  5 root root   94 Feb  8 12:23 ..
drwxrwxrwx  2 root root   51 Feb  8 12:22 .datalad
drwxrwxrwx  9 root root  185 Feb  8 12:23 .git
-rw-rw-rw-  1 root root   55 Feb  8 12:22 .gitattributes
-rw-rw-rw-  1 root root  806 Feb  8 12:22 .gitmodules
drwxrwxrwx  5 root root  589 Feb  8 12:22 tpl-MNI152Lin
drwxrwxrwx  4 root root 1.7K Feb  8 12:22 tpl-MNI152NLin2009cAsym
drwxrwxrwx  5 root root  976 Feb  8 12:22 tpl-NKI
drwxrwxrwx  4 root root 1022 Feb  8 12:22 tpl-OASIS30ANTs
drwxrwxrwx  4 root root  689 Feb  8 12:22 tpl-PNC
drwxrwxrwx  5 root root  235 Feb  8 12:22 tpl-fMRIPrep
drwxrwxrwx  4 root root 1.2K Feb  8 12:22 tpl-fsLR
drwxrwxrwx  5 root root 1.5K Feb  8 12:23 tpl-fsaverage

/opt/templateflow also has the right permissions.

Singularity poldracklab_fmriprep_1.3.0-1-2019-02-08-1b5ad521ee2a.simg:/opt/templateflow> datalad get tpl-OASIS30ANTs/*
Total (0 ok, 9 failed out of 9):   0%|                                                                                                                                                | 0.00/18.4M [00:03<?, ?B/s][WARNING] Running get resulted in stderr output: download failed: .git/annex/tmp/URL-s255798--https&c%%files.osf.io%v1%resourc-9d3f2d9def2df5e2c10c61e8ae34204f: openBinaryFile: permission denied (Read-only file system)
git-annex: .git/annex/transfer/failed: createDirectory: permission denied (Read-only file system)
download failed: .git/annex/tmp/URL-s726641--https&c%%files.osf.io%v1%resourc-b8c123a75a0b1321a310008fcec889de: openBinaryFile: permission denied (Read-only file system)
git-annex: .git/annex/transfer/failed: createDirectory: permission denied (Read-only file system)
download failed: .git/annex/tmp/URL-s446641--https&c%%files.osf.io%v1%resourc-959273ae38edfdca40ce65784835c21f: openBinaryFile: permission denied (Read-only file system)
git-annex: .git/annex/transfer/failed: createDirectory: permission denied (Read-only file system)
download failed: .git/annex/tmp/URL-s6205417--https&c%%files.osf.io%v1%resourc-75376237d0e44a6092644c46ea03ed7b: openBinaryFile: permission denied (Read-only file system)
git-annex: .git/annex/tra... 
                                                                                                                                                                                                                  [ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-SCGM_probseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-SCGM_probseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-BS_probseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-BS_probseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-CSF_probseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-CSF_probseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-6_dseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-6_dseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-CBM_probseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-CBM_probseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-CGM_probseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-CGM_probseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-4_dseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-4_dseg.nii.gz (file) [from web...]
[ERROR  ] from web... [get(/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-WM_probseg.nii.gz)] 
get(error): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-WM_probseg.nii.gz (file) [from web...]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/CHANGES (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/template_description.json (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/template_sample.tsv (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_T1w.nii.gz (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-4_dseg.tsv (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-6_dseg.tsv (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumExtraction_mask.nii.gz (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-brain_T1w.nii.gz (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-brain_mask.nii.gz (file) [already present]
get(notneeded): /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-brain_probseg.nii.gz (file) [already present]
action summary:
  get (error: 9, notneeded: 10)

pinging @yarikoptic just in case he may spot where we are doing it wrong.

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 8, 2019

I guess the filesystem layer of a Singularity image can't be written on top of.

@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

So move templateflow out of /opt ?

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 8, 2019

I think it means we can't use pre-downloaded templates in Singularity.

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

Iirc you could specify overlay and then you would be able to

@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

A quick workaround while we figure this out that worked for me:

  • datalad install -r ... (somewhere you will have access to)
    run singularity binding that path to /opt/templateflow
@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

Or image could just come with them, ie get them while generating it or that is too heavy?

@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

We are doing that for the most standard templates at this moment.

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 8, 2019

TBH it's not really clear why it's trying to do anything with git, when the files are already downloaded.

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

while I am fetching the monster of the fmriprep image to check (is "docker pull poldracklab/fmriprep" enough or should I give some tag?) -- from the log it seems that some are there and some (those failing like
/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_label-WM_probseg.nii.gz ) not there.

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

yeap - file is not there:

$> docker run -it --rm --entrypoint bash poldracklab/fmriprep    
root@6d325eccc7eb:/tmp# ls -l /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
lrwxrwxrwx 1 root root 182 Feb  8 01:10 /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz -> .git/annex/objects/X2/fq/URL-s255798--https&c%%files.osf.io%v1%resourc-9d3f2d9def2df5e2c10c61e8ae34204f/URL-s255798--https&c%%files.osf.io%v1%resourc-9d3f2d9def2df5e2c10c61e8ae34204f
root@6d325eccc7eb:/tmp# ls -lL /opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
ls: cannot access '/opt/templateflow/tpl-OASIS30ANTs/tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz': No such file or directory

as some others

root@6d325eccc7eb:/tmp# cd /opt/templateflow/tpl-OASIS30ANTs/
root@6d325eccc7eb:/opt/templateflow/tpl-OASIS30ANTs# git annex list
here
|origin
||web
|||bittorrent
||||
X_X_ tpl-OASIS30ANTs_res-01_T1w.nii.gz
__X_ tpl-OASIS30ANTs_res-01_desc-4_dseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_desc-6_dseg.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_desc-BrainCerebellumExtraction_mask.nii.gz
__X_ tpl-OASIS30ANTs_res-01_desc-BrainCerebellumRegistration_mask.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_desc-brain_T1w.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_desc-brain_mask.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-BS_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-CBM_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-CGM_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-CSF_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-SCGM_probseg.nii.gz
__X_ tpl-OASIS30ANTs_res-01_label-WM_probseg.nii.gz
X_X_ tpl-OASIS30ANTs_res-01_label-brain_probseg.nii.gz
@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

Yep, the fastest solution would be to pull everything (check http://singularity.lbl.gov/archive/docs/v2-2/create-image#read-only-vs-read-write).

@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 9, 2019

Okay, I'm going to close this via bca40d1

Singularity users willing to use other templates not pre-loaded in the image will need to bind a writable folder in the host to /opt/templateflow.

@oesteban

This comment has been minimized.

Copy link
Contributor

commented Feb 9, 2019

Just checked the new singularity image does not attempt to download anything. Will release a patched version soon.

@oesteban oesteban closed this Feb 9, 2019

@effigies effigies reopened this Feb 13, 2019

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

Okay, just to return to this, @yarikoptic, assuming the files are present, should datalad be calling git-annex when the files exist? I can't find the source code for templateflow, but I assume that it's making some call to datalad.

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2019

who knows @effigies ;-) probably not since for get we first talk to annex to discover what files we need to download. If no files needed, should be all good.
But related -- I did run previously into cases where git annex (and git?) were trying to lock repository for seemingly read-only operation. Ah here -- https://git-annex.branchable.com/bugs/impossible_to_perform___34__read-only__34___git_annex_info_without_write_permissions/ ... not sure if we actually accounted for that yet, but I do not think it should be triggered in this case.

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 13, 2019

but what lead to reopening -- you got permission denied again @effigies ? isn't templateflow a part of fmriprep? (I saw it there IIRC)

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

TemplateFlow is kind of its own thing, and I'm trying to figure out how it works. Oscar's fix did not resolve issues for users, so I'm trying to figure out why we're getting the error seen. There's currently a full traceback here: https://gist.github.com/Gilles86/75c3b848f1973d947f0838a1020f7f76#file-fmriprep-1-3-0-post1

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

The problem seems to be the lockfile, which apparently gets held even for read-only operations. That may be what's being described in your post, @yarikoptic. Do you understand what he means by annex.merge-annex-branches=false and where to set that? I could try manually setting that in the Dockerfile.

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

Oh, I see git config options.

@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2019

Yes, please try! If works - great. Regardless, we should workaround on the level of datalad. For find here we definitely not need merging annex branch info.
Another workaround for you at the frmriprep would probably to remove remote after installing dataset... But that is not pretty

@effigies

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 14, 2019

I was able to reproduce the bug with a new singularity image built from poldracklab/fmriprep:1.3.0.post1 and it did not appear with a new image built from poldracklab/fmriprep:tfpatch. I will re-run tomorrow morning, in case there's a minimum delay before git-annex attempts to hold the lockfile.

yarikoptic added a commit to yarikoptic/datalad that referenced this issue Feb 14, 2019

BF: set annex.merge-annex-branches=false for invocations not requirin…
…g updated remote information

git-annex pretty much for any command which could potentially benefit
from updated availability information present in updated remote git-annex
branches will try to merge those first.  But if that fails, the whole call
fails, even if what we are asking does not require updated remote information.

Use cases where we ran into it already:
-  datalad ls -L  of a dataset owned by someone else
-  datalad get files   whenever files load is already there but partition
      is not writeable (singularity image, see poldracklab/fmriprep#1500 (comment)

So with this change we set -c annex.merge-annex-branches=false for those
git annex invocations where I think we do not really need any git-annex
branch being merged since we care only about local stuff.

This commit comes without a test since I have failed to simulate the
failing scenario without sudo chown someonelse -R testrepo . We could
probably do that on Travis, but I thought first to check if there
could be better ideas
@yarikoptic

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2019

fix on datalad side is WiP: datalad/datalad#3164

yarikoptic added a commit to yarikoptic/datalad that referenced this issue Feb 14, 2019

BF: set annex.merge-annex-branches=false for invocations not requirin…
…g updated remote information

git-annex pretty much for any command which could potentially benefit
from updated availability information present in updated remote git-annex
branches will try to merge those first.  But if that fails, the whole call
fails, even if what we are asking does not require updated remote information.

Use cases where we ran into it already:
-  datalad ls -L  of a dataset owned by someone else
-  datalad get files   whenever files load is already there but partition
      is not writeable (singularity image, see poldracklab/fmriprep#1500 (comment)

So with this change we set -c annex.merge-annex-branches=false for those
git annex invocations where I think we do not really need any git-annex
branch being merged since we care only about local stuff.

This commit comes without a test since I have failed to simulate the
failing scenario without sudo chown someonelse -R testrepo . We could
probably do that on Travis, but I thought first to check if there
could be better ideas

yarikoptic added a commit to yarikoptic/datalad that referenced this issue Feb 14, 2019

BF: set annex.merge-annex-branches=false for invocations not requirin…
…g updated remote information

pretty much for any command which could potentially benefit
from updated availability information present in updated remote git-annex
branches git-annex will try to merge those first.  But if that fails, the whole call
fails, even if what we are asking does not require updated remote information.

Use cases where we ran into it already:
-  datalad ls -L  of a dataset owned by someone else
-  datalad get files   whenever files load is already there but partition
      is not writeable (singularity image, see poldracklab/fmriprep#1500 (comment)

So with this change we set -c annex.merge-annex-branches=false for those
git annex invocations where I think we do not really need any git-annex
branch being merged since we care only about local stuff.

This commit comes without a test since I have failed to simulate the
failing scenario without sudo chown someonelse -R testrepo . We could
probably do that on Travis, but I thought first to check if there
could be better ideas
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.