Template submission failed

[jdiedrichsen]

Dear Oscar,

I am trying to contribute the SUIT (cerebellar only) template to templateflow, following the submission instructions:
https://www.templateflow.org/contributing/submission/

The file upload does work - however, there seems to be a conflict with something uploaded earlier.

I'm not sure how to resolve this....

best,

Joern

tfmgr add tpl-SUIT --osf-user *******
OSF password: 
Has a RRID (research resource ID) already been assigned? []: No
Preparing Pull-Request (wd=/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74).
Cloning into 'templateflow'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (2/2), done.
remote: Total 15 (delta 0), reused 2 (delta 0), pack-reused 13
Receiving objects: 100% (15/15), 44.62 MiB | 20.54 MiB/s, done.
Resolving deltas: 100% (1/1), done.
create(error): /private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT (dataset) [collision with content in parent dataset at /private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow: ['/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/LICENSE', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/README.md', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/template_description.json', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT.spec', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-image_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-point_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-image_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-point_xfm.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_res-1_T1w.nii', '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii']]
Traceback (most recent call last):
  File "/opt/miniconda3/bin/tfmgr", line 8, in <module>
    sys.exit(cli())
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/miniconda3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.9/site-packages/tfmanager/cli.py", line 221, in add
    dl.create(
  File "/opt/miniconda3/lib/python3.9/site-packages/datalad/interface/base.py", line 773, in eval_func
    return return_func(*args, **kwargs)
  File "/opt/miniconda3/lib/python3.9/site-packages/datalad/interface/base.py", line 763, in return_func
    results = list(results)
  File "/opt/miniconda3/lib/python3.9/site-packages/datalad_next/patches/interface_utils.py", line 287, in _execute_command_
    raise IncompleteResultsError(
datalad.support.exceptions.IncompleteResultsError: Command did not complete successfully. 1 failed:
[{'action': 'create',
  'message': ('collision with content in parent dataset at %s: %s',
              '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow',
              ['/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/LICENSE',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/README.md',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/template_description.json',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT.spec',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-image_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin2009cSymC_mode-point_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-image_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_from-MNI152NLin6AsymC_mode-point_xfm.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_res-1_T1w.nii',
               '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii']),
  'path': '/private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow/tpl-SUIT',
  'refds': None,
  'status': 'error',
  'type': 'dataset'}]

[oesteban]

Sorry, sorry, sorry -- I've been up to my eyeballs and totally forgot about this :(

First, I'm going to move this comment to the template manager. It looks like an error on your computer's side.

However, I don't want to keep you stuck with this, so if you zip up the template and send it to me, I'll make sure to publish it before Tuesday.

Again, thanks a lot for your incredible patience (since September, when you first contacted me).

[oesteban]

Okay, so the installation is failing at

python-manager/tfmanager/cli.py

Lines 221 to 226 in f4c596a

	dl.create(
	path=f"tpl-{template_id}",
	cfg_proc="text2git",
	initopts={"initial-branch": "main"},
	description=metadata["Name"],
	)

meaning, it is trying to create a new datalad dataset but failing. It seems that your templateflow super-dataset at /private/var/folders/vh/tsrrmndj3kj8fm3m4p51k3nc0000gn/T/tmpzezi_c74/templateflow has already a tpl-SUIT installation. I'm not sure why this may be happening, except if you are not using the "official" templateflow super-dataset.

Something's off, since your error back in September was slightly different:

jdiedrichsen@diedrichsenofficemac templateflow % tfmgr add tpl-SUIT --osf-user ******
OSF password: 
Has a RRID (research resource ID) already been assigned? []: no
Preparing Pull-Request (wd=/var/folders/sz/7lqgfrln5738xc_hktty2sb00000gp/T/tmpamzeer03).
Cloning into 'templateflow'...
warning: Could not find remote branch tpl-intake to clone.
fatal: Remote branch tpl-intake not found in upstream origin
/bin/sh: hub: command not found
Traceback (most recent call last):
  File "/Users/jdiedrichsen/opt/miniconda3/bin/tfmgr", line 8, in <module>
    sys.exit(cli())
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/tfmanager/cli.py", line 205, in add
    run_command(
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/site-packages/tfmanager/io.py", line 29, in run_command
    proc = run(cmd, capture_output=capture_output, shell=True, env=env, cwd=cwd,)
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/subprocess.py", line 489, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/jdiedrichsen/opt/miniconda3/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/sz/7lqgfrln5738xc_hktty2sb00000gp/T/tmpamzeer03/templateflow'

[oesteban]

By reading the prompt, it looks like you are executing tfmgr under an existing templateflow repo. If that repo is a fork (I know you did a fork because you invited me to it a while ago), it is possible that git is getting confused.

[jdiedrichsen]

Maybe I am just doing something wrong.

Here is what I did:

• I created a new repository called templateflow, only with tpl-SUIT in it- I did NOT fork or clone the main templateflow repro (maybe that's the problem?)
• I then called tfmgr add tpl-SUIT
• When I got the first error (in September), I create a branch tpl-intake manually, which avoided the first error, but gave me the second

I also tried to call tfmgr without having a dedicated repro, but that did not work either.

Let me know what the correct approach is - and I give it another try from here.

Joern

[jdiedrichsen]

I tried a couple of more variations, but couldn't make it work - did I have to fork from templateflow?

A zipped version of the template (46MB) is a bit big to upload here - so here is a dropbox link:

https://www.dropbox.com/scl/fi/aqqysabl0esns730mzlap/tpl-SUIT.zip?rlkey=rnke2wosbmr0g74g2dthp7lyv&dl=0

if you could submit from your side, that would be great!

Joern

[oesteban]

Hi @jdiedrichsen, I have confirmed that the templateflow manager is quite broken.

I think your errors come from two assumptions:

• You will only once ever upload a template to templateflow (okay, fine, this was your first)
• The root templateflow repository does not change since you fork it the first time. I'm afraid this is what happened in your case (although I will need to investigate it deeper).

However, even if you had been more lucky and got past the point where it broke for you, then you would've hit several authentication issues. Datalad has changed substantially since I last updated the manager and now things are not working.

In the end, I had to manually push the submission process. I have taken a few liberties:

• I removed the transforms files because they were 3D images with a gradient. It looks they could be a 4x4 matrix instead. I wanted to confirm this point.
• I added two atlases: Diedrichsen2009 and Buckner2011. I hope that is okay with you :)
  My plan is to add the other two or three you have in your repo.

I just tested and things seem to work:

• The online browser shows SUIT:
• The command line lists the files in the template:

  $ templateflow --version
  TemplateFlow Python Client 24.2.0.dev49+gb1f2eb2a.d20240413
  $ templateflow update
  $ templateflow ls SUIT
  /data/templateflow/tpl-SUIT/CHANGES
  /data/templateflow/tpl-SUIT/LICENSE
  /data/templateflow/tpl-SUIT/README.md
  /data/templateflow/tpl-SUIT/template_description.json
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_dseg.json
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.label.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.tsv
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_stat-confidence_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.label.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.tsv
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_stat-confidence_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.json
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.label.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.tsv
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_probseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_T1w.nii.gz
  

• And they seem correctly fetched if we request them

  $ templateflow get SUIT
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.label.gii
  100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.67k/9.67k [00:00<00:00, 3.57MB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.nii.gz
  100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51.7k/51.7k [00:00<00:00, 554kB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_stat-confidence_dseg.nii.gz
  100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 776k/776k [00:00<00:00, 1.41MB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.label.gii
  100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10.6k/10.6k [00:00<00:00, 3.84MB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.nii.gz
  100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44.3k/44.3k [00:00<00:00, 475kB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_probseg.nii.gz
  100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.77M/1.77M [00:01<00:00, 1.73MB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_flat.surf.gii
  100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 591k/591k [00:00<00:00, 1.05MB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_sulc.shape.gii
  100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 147k/147k [00:00<00:00, 534kB/s]
  Downloading https://templateflow.s3.amazonaws.com/tpl-SUIT/tpl-SUIT_T1w.nii.gz
  100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 385k/385k [00:00<00:00, 728kB/s]
  /data/templateflow/tpl-SUIT/CHANGES
  /data/templateflow/tpl-SUIT/LICENSE
  /data/templateflow/tpl-SUIT/README.md
  /data/templateflow/tpl-SUIT/template_description.json
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_dseg.json
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.label.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_dseg.tsv
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-7n_stat-confidence_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.label.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_dseg.tsv
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Buckner2011_seg-17n_stat-confidence_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.json
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.label.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_dseg.tsv
  /data/templateflow/tpl-SUIT/tpl-SUIT_atlas-Diedrichsen2009_probseg.nii.gz
  /data/templateflow/tpl-SUIT/tpl-SUIT_flat.surf.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_sulc.shape.gii
  /data/templateflow/tpl-SUIT/tpl-SUIT_T1w.nii.gz
  

I'm going to close this issue and open new ones about the submission process. For questions/problems/future topics relating SUIT (e.g., adding the remainder of atlases, or fixing things I didn't get correctly), let's use the template repository at https://github.com/templateflow/tpl-SUIT (I've enabled the issues tab to keep track of these)

Thanks a lot for your patience :)