add site data ucdavis #77

alexfoias · 2021-03-24T22:15:32Z

Curently missing sex, age, date of scan.

Here is a QC of defacing ucdavis_spineGeneric_QC.zip

alexfoias · 2021-03-25T13:17:20Z

We didn't get the date_of_scan info. I think we can go without it.

jcohenadad · 2021-04-05T19:55:14Z

participants.tsv

@@ -227,6 +227,13 @@ sub-ubc03	F	24	2019-06-06	ubc	"University of British Columbia (UBC), Vancouver,
 sub-ubc04	F	25	2019-06-06	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
 sub-ubc05	M	24	2019-06-07	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
 sub-ubc06	M	23	2019-06-17	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
+sub-ucdavis01	F	32	-	ucdavis	"UC Davis Health, California, US"	Siemens	Magentom Vida	HeadNeck_64_CS	syngo MR XA20	Allan R. Martin


typo: Magentom --> Magnetom

also, no need to mention "Magnetom". We don't do it for Prisma, Skyra, etc.

Fixed in 8143f98

jcohenadad · 2021-04-05T19:56:53Z

participants.tsv

@@ -227,6 +227,13 @@ sub-ubc03	F	24	2019-06-06	ubc	"University of British Columbia (UBC), Vancouver,
 sub-ubc04	F	25	2019-06-06	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
 sub-ubc05	M	24	2019-06-07	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
 sub-ubc06	M	23	2019-06-17	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
+sub-ucdavis01	F	32	-	ucdavis	"UC Davis Health, California, US"	Siemens	Magentom Vida	HeadNeck_64_CS	syngo MR XA20	Allan R. Martin
+sub-ucdavis02	F	56	-	ucdavis	"UC Davis Health, California, US"	Siemens	Magentom Vida	HeadNeck_64_CS	syngo MR XA21	Allan R. Martin


Replace HeadNeck_64_CS --> HeadNeck_64 (let's try to standardize a bit...)

Fixed in 8143f98

jcohenadad · 2021-04-05T19:57:45Z

participants.tsv

@@ -227,6 +227,13 @@ sub-ubc03	F	24	2019-06-06	ubc	"University of British Columbia (UBC), Vancouver,
 sub-ubc04	F	25	2019-06-06	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
 sub-ubc05	M	24	2019-06-07	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
 sub-ubc06	M	23	2019-06-17	ubc	"University of British Columbia (UBC), Vancouver, Canada"	Philips	Ingenia-ElitionX	16chHeadNeck+Posterior	R5.5	"A.V. Dvorak, E. MacMillan, S. Kolind"
+sub-ucdavis01	F	32	-	ucdavis	"UC Davis Health, California, US"	Siemens	Magentom Vida	HeadNeck_64_CS	syngo MR XA20	Allan R. Martin
+sub-ucdavis02	F	56	-	ucdavis	"UC Davis Health, California, US"	Siemens	Magentom Vida	HeadNeck_64_CS	syngo MR XA21	Allan R. Martin
+sub-ucdavis03	M	41	-	ucdavis	"UC Davis Health, California, US"	Siemens	Magentom Vida	HeadNeck_64_CS	syngo MR XA22	Allan R. Martin


"syngo MR XA22" -> the last number is different across subjects... this is suspicious...

Fixed in 8143f98

alexfoias · 2021-04-12T19:06:05Z

@jcohenadad I have updated the participants.tsv

jcohenadad · 2021-04-12T19:27:16Z

oopsi, any idea what's going on?
https://github.com/spine-generic/data-multi-subject/pull/77/checks?check_run_id=2326970382#step:8:2770

alexfoias · 2021-04-12T19:58:45Z

@kousu The build fails. The error looks something like

get sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (not available) 
  No other repository is known to contain the file.

  (Note that these git remotes have annex-ignore set: origin)
failed

Did I do something wrong?

kousu · 2021-04-12T21:05:51Z

Probably that file didn't get upload to Amazon correctly. I'm investigating.

kousu · 2021-04-12T21:12:44Z

Possibly because git-annex keeps its metadata in a separate branch (neuropoly/data-management#67). This was submitted from a fork which means the git-annex branch is isolated over in https://github.com/alexfoias/data-multi-subject/ and this repo isn't seeing it. Maybe. I still need to wait for things to sync fully before I can check for sure.

I'll see if I can figure out how to tweak the CI script to be able to deal with forks. But because of this separate branch problem git-annex and the Github GUI are never going to be fully compatible.

kousu · 2021-04-12T21:34:34Z

It's a simpler, but related, problem: @alexfoias your local git-annex branch hasn't been synchronized to github:

$ git remote add af https://github.com/alexfoias/data-multi-subject/
$ git fetch af
remote: Enumerating objects: 9058, done.
remote: Counting objects: 100% (9058/9058), done.
remote: Compressing objects: 100% (1984/1984), done.
remote: Total 16921 (delta 5816), reused 8389 (delta 5709), pack-reused 7863
Receiving objects: 100% (16921/16921), 1.55 MiB | 3.50 MiB/s, done.
Resolving deltas: 100% (6918/6918), completed with 2004 local objects.
From https://github.com/alexfoias/data-multi-subject
 * [new branch]        af/add_ucdavis                       -> af/af/add_ucdavis
 * [new branch]        af/update_participants_tsv           -> af/af/update_participants_tsv
 * [new branch]        git-annex                            -> af/git-annex
 * [new branch]        master                               -> af/master
 * [new branch]        renelabounek-master                  -> af/renelabounek-master
 * [new branch]        synced/af/csfseg_json                -> af/synced/af/csfseg_json
 * [new branch]        synced/git-annex                     -> af/synced/git-annex
 * [new branch]        synced/jca/17-defacing               -> af/synced/jca/17-defacing
 * [new branch]        synced/jca/21-travis                 -> af/synced/jca/21-travis
 * [new branch]        synced/jca/7-check-data-consistency  -> af/synced/jca/7-check-data-consistency
 * [new branch]        synced/jca/derivatives-r20200830-mts -> af/synced/jca/derivatives-r20200830-mts
 * [new branch]        synced/jca/participants              -> af/synced/jca/participants
 * [new branch]        synced/jca/readme                    -> af/synced/jca/readme
 * [new branch]        synced/jca/reupload-disc-labels      -> af/synced/jca/reupload-disc-labels
 * [new branch]        synced/jca/update-derivatives        -> af/synced/jca/update-derivatives
 * [new branch]        synced/master                        -> af/synced/master
[kousu@requiem data-multi-subject]$ git annex whereis  sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz
(merging af/git-annex into git-annex...)
(recording state in git...)

  Remote af not usable by git-annex; setting annex-ignore

  https://github.com/alexfoias/data-multi-subject//config download failed: Not Found
git-annex: sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz not found
git-annex: whereis: 1 failed
$ git log --pretty=short af/git-annex^^..af/git-annex
commit 3cc730e6fac61f41c969b60af0d62d794250fe3b (af/synced/git-annex, af/git-annex)
Author: Alexandru Foias <afoias@polymtl.ca>

    update

commit 39d9c91f1bacc5fe9452028a2b8c4bc344438de7
Author: Alexandru Foias <afoias@polymtl.ca>

    update
$

(notice the timestamps)

You need to do

git annex sync

More directly you can do

git push origin git-annex

which is what I am thinking more and more I want to recommend.

alexfoias · 2021-04-13T11:34:58Z

@kousu I did this:

[13/04/21 7:33:29] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git push origin git-annex
Enumerating objects: 290, done.
Counting objects: 100% (290/290), done.
Delta compression using up to 12 threads
Compressing objects: 100% (237/237), done.
Writing objects: 100% (261/261), 112.78 KiB | 6.63 MiB/s, done.
Total 261 (delta 53), reused 1 (delta 0)
remote: Resolving deltas: 100% (53/53), completed with 4 local objects.
To https://github.com/alexfoias/data-multi-subject.git
   3cc730e6..c7283b1c  git-annex -> git-annex
[13/04/21 7:33:33] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex sync 
commit 
On branch af/add_ucdavis
Your branch is up to date with 'origin/af/add_ucdavis'.

nothing to commit, working tree clean
ok
pull origin 
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 1), reused 3 (delta 1), pack-reused 0
Unpacking objects: 100% (5/5), done.
From https://github.com/alexfoias/data-multi-subject
   8143f98b..7f58747b  af/add_ucdavis -> origin/af/add_ucdavis

Updating 8143f98b..7f58747b
Fast-forward
 README.md | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
ok
push origin 
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/alexfoias/data-multi-subject.git
   3cc730e6..c7283b1c  git-annex -> synced/git-annex
 * [new branch]        af/add_ucdavis -> synced/af/add_ucdavis
ok
[13/04/21 7:34:05] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git push origin git-annex
Everything up-to-date

kousu · 2021-04-13T18:59:31Z

@alexfoias Problem 2 seems to be that the files have not been uploaded to Amazon

$ git branch
* af/add_ucdavis
master
$ git annex whereis sub-ucdavis06/
whereis sub-ucdavis06/anat/sub-ucdavis06_T1w.nii.gz (1 copy) 
  	56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_T2star.nii.gz (1 copy) 
  	56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_T2w.nii.gz (1 copy) 
  	56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_acq-MToff_MTS.nii.gz (1 copy) 
  	56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS.nii.gz (1 copy) 
  	56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok
whereis sub-ucdavis06/dwi/sub-ucdavis06_dwi.nii.gz (1 copy) 
  	56bbd6c5-a147-4940-bf73-212f50841743 -- alex@NeuroPoly-MacBook-Pro.local:~/data/data-multi-subject
ok

Can you try this?

git annex dead here
git annex sync --content amazon
git annex sync

alexfoias · 2021-04-13T19:28:47Z

@kousu Still doesn't work https://gist.github.com/alexfoias/bdf8f28c3e2adbe8048cc3ce4aaf6736

kousu · 2021-04-13T20:26:08Z

@alexfoias oops, you need to load your Amazon password:

copy sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (checking amazon...) (to amazon...) 

  Set both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to use S3

  No S3 credentials configured
failed

You need to run

export AWS_ACCESS_KEY_ID="..." AWS_SECRET_ACCESS_KEY="..."

You can put those in your ~/.profile (or, ~/.zprofile? You're on the new macOS right? is that the right path now?).

I wonder if we can script this to make this easier. Personally, I have my AWS keys in my password manager so what I actually do is

AWS_ACCESS_KEY_ID="$(pass work/neuropoly/aws | grep username: | awk '{print $2}')" AWS_SECRET_ACCESS_KEY="$(pass work/neuropoly/aws | head -n 1)" git annex sync --content amazon

which prompts me to unlock my password manager.

I probably should have told you git annex copy --to amazon. I'm sorry you had to wait for the downloads before you could upload, too.

(the docs are here)

kousu · 2021-04-13T20:43:23Z

Turns out @alexfoias had never had AWS access in the first place. I've granted him write-access to it.

alexfoias · 2021-04-16T14:18:03Z

@kousu How do I have to add the AWS credentials ? Do I just have to set the env var export AWS_ACCESS_KEY_ID="..." AWS_SECRET_ACCESS_KEY="..." ?

kousu · 2021-04-16T15:18:26Z

@alexfoias yes, exactly. Just paste them into your terminal and press enter!

You can also make them more permanent with these suggestions: #77 (comment)

alexfoias · 2021-04-16T15:55:30Z

After exporting the AWS credentials:

[16/04/21 11:53:31] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex dead here
dead here (recording state in git...)
ok
(recording state in git...)
[16/04/21 11:53:33] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex sync --content amazon
commit 
On branch af/add_ucdavis
Your branch is up to date with 'origin/af/add_ucdavis'.

nothing to commit, working tree clean
ok
copy sub-ucdavis01/anat/sub-ucdavis01_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/anat/sub-ucdavis01_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/anat/sub-ucdavis01_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/anat/sub-ucdavis01_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/anat/sub-ucdavis01_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/anat/sub-ucdavis01_acq-T1w_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/dwi/sub-ucdavis01_run-01_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/dwi/sub-ucdavis01_run-02_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis01/dwi/sub-ucdavis01_run-03_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/anat/sub-ucdavis02_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/anat/sub-ucdavis02_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/anat/sub-ucdavis02_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/anat/sub-ucdavis02_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/anat/sub-ucdavis02_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/anat/sub-ucdavis02_acq-T1w_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/dwi/sub-ucdavis02_run-01_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/dwi/sub-ucdavis02_run-02_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis02/dwi/sub-ucdavis02_run-03_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis03/anat/sub-ucdavis03_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis03/anat/sub-ucdavis03_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis03/anat/sub-ucdavis03_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis03/anat/sub-ucdavis03_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis03/anat/sub-ucdavis03_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis03/dwi/sub-ucdavis03_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/anat/sub-ucdavis04_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/anat/sub-ucdavis04_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/anat/sub-ucdavis04_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/anat/sub-ucdavis04_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/anat/sub-ucdavis04_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/anat/sub-ucdavis04_acq-T1w_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/dwi/sub-ucdavis04_run-01_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/dwi/sub-ucdavis04_run-02_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis04/dwi/sub-ucdavis04_run-03_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis05/anat/sub-ucdavis05_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis05/anat/sub-ucdavis05_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis05/anat/sub-ucdavis05_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis05/anat/sub-ucdavis05_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis05/anat/sub-ucdavis05_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis05/dwi/sub-ucdavis05_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis06/anat/sub-ucdavis06_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis06/anat/sub-ucdavis06_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis06/anat/sub-ucdavis06_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis06/anat/sub-ucdavis06_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis06/anat/sub-ucdavis06_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis06/dwi/sub-ucdavis06_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis07/anat/sub-ucdavis07_T1w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis07/anat/sub-ucdavis07_T2star.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis07/anat/sub-ucdavis07_T2w.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis07/anat/sub-ucdavis07_acq-MToff_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS.nii.gz (checking amazon...) (to amazon...) 
ok                                
copy sub-ucdavis07/dwi/sub-ucdavis07_dwi.nii.gz (checking amazon...) (to amazon...) 
ok                                
(recording state in git...)
[16/04/21 11:54:35] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git annex sync
commit 
On branch af/add_ucdavis
Your branch is up to date with 'origin/af/add_ucdavis'.

nothing to commit, working tree clean
ok
pull origin 
ok
push origin 
Enumerating objects: 310, done.
Counting objects: 100% (310/310), done.
Delta compression using up to 12 threads
Compressing objects: 100% (138/138), done.
Writing objects: 100% (161/161), 12.54 KiB | 1.39 MiB/s, done.
Total 161 (delta 57), reused 0 (delta 0)
remote: Resolving deltas: 100% (57/57), completed with 55 local objects.
To https://github.com/alexfoias/data-multi-subject.git
   2501e150..9524c416  git-annex -> synced/git-annex
ok

alexfoias · 2021-04-16T16:09:50Z

@kousu Houston we have a problem :(

kousu · 2021-04-16T16:11:57Z

@alexfoias git push origin git-annex?

kousu · 2021-04-16T16:15:31Z

Yeah looks like that's the problem:

$ git fetch af
remote: Enumerating objects: 8, done.
remote: Counting objects: 100% (8/8), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 8 (delta 2), reused 3 (delta 2), pack-reused 0
Unpacking objects: 100% (8/8), 9.47 KiB | 111.00 KiB/s, done.
From https://github.com/alexfoias/data-multi-subject
   7f58747b5..dc66d8e76  af/add_ucdavis -> af/af/add_ucdavis

It should say

   7f58747b5..dc66d8e76  af/add_ucdavis -> af/af/add_ucdavis
   9524c4169..something  af/git-annex -> af/git-annex

alexfoias · 2021-04-16T16:17:54Z

[16/04/21 12:17:31] (base) alex@NeuroPoly-MacBook-Pro data-multi-subject % git push origin git-annex
Everything up-to-date

kousu · 2021-04-16T16:17:56Z

Oh no, I'm not sure about that.

kousu · 2021-04-16T16:21:10Z

It looks good on my local copy:

$ git annex whereis sub-ucdavis07
whereis sub-ucdavis07/anat/sub-ucdavis07_T1w.nii.gz (1 copy) 
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s4484346--9b027c074735c3d2d69f046ebaec5831c4e6fc8959a3a64070428ebd75a62fd9.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_T2star.nii.gz (1 copy) 
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s1765026--7fb7243348a5bf2c8994cd383848aa3c279ee072701ba56bf43f50b8fb06efbf.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_T2w.nii.gz (1 copy) 
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s6620793--78252c6bb18e92298b2b7ec79d4ec5d441750790d7849c0dc174a78a95e669a3.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_acq-MToff_MTS.nii.gz (1 copy) 
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s1355554--69fe2b3b512b64b0bfcedfddfe5f8485725db699142010898c20aa2b3c63c0af.nii.gz
ok
whereis sub-ucdavis07/anat/sub-ucdavis07_acq-MTon_MTS.nii.gz (1 copy) 
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s1294757--043e6b105eec24f64f1dd72b221d7b58569f02bcc9bed42e99795d99a5dd26d2.nii.gz
ok
whereis sub-ucdavis07/dwi/sub-ucdavis07_dwi.nii.gz (1 copy) 
  	5a5447a8-a9b8-49bc-8276-01a62632b502 -- [amazon]

  amazon: https://data-multi-subject---spine-generic---neuropoly.s3.ca-central-1.amazonaws.com/SHA256E-s2348392--12a62969d9a30c726f5471a104c7e42817e3e1509bc5171bbb420e3f4b61fa4f.nii.gz
ok

so I don't yet know what's wrong with Github. But my money is on Github not downloading the git-annex branch out of your fork.

kousu · 2021-04-16T16:24:09Z

Yes, I think that's it: https://github.com/spine-generic/data-multi-subject/pull/77/checks?check_run_id=2363833281#step:8:123

Run git fetch --depth=1 origin git-annex:git-annex # CI does a shallow checkout, so it is missing this important branch
From https://github.com/spine-generic/data-multi-subject
 * [new branch]      git-annex  -> git-annex
 * [new branch]      git-annex  -> origin/git-annex
init  (scanning for unlocked files...)
  Remote origin not usable by git-annex; setting annex-ignore

so it's fetching the git-annex from this repo not your repo. We'll have to patch the CI script to fix that. Awkward.

kousu · 2021-04-16T16:30:04Z

I'm going to mess with your branch a bit to figure this out.

kousu · 2021-04-16T17:00:08Z

This is a classic case of neuropoly/data-management#67

kousu · 2021-04-16T18:20:40Z

I got it working. I'm going to merge now.

sub-ucdavis04, sub-ucdavis05, sub-ucdavis06, sub-ucdavis07

kousu · 2021-04-16T19:03:00Z

😥

jcohenadad · 2021-04-16T19:10:31Z

thank you @alexfoias and @kousu. These painful experiences add to cons of git-annex

kousu · 2021-04-16T19:12:16Z

It's not fully git-annex's fault. It's just not fully compatible with the Github workflow. Equally much you could say "add to the cons of Github".

kousu · 2021-04-16T19:18:58Z

https://github.com/spine-generic/data-multi-subject/actions/runs/756637528 😓

I predicted this just now:

data-multi-subject/.github/workflows/validator.yml

Line 56 in 46df262

    
           # this might be subtly buggy since it is testing the remote version, not the merged version

okay I'll see what I can do about it.

Right, of course, it's literally the same bug:

(recording state in git...)
git-annex: get: 51 failed
Error: Process completed with exit code 1.

Clicking the 'merge' button doesn't also merge the git-annex branch.

What to do about that?

Never use forks ever? Or never merge git-annex changes on the command line; but we have branch protection turned on so we cannot merge from the command line, it has to be done through the Github GUI? Or make sure to make associated PRs for the git-annex branch?

kousu · 2021-04-16T19:23:48Z

I'm trying my third strategy this time: #84. Can one of you approve it @jcohenadad @alexfoias ?

jcohenadad · 2021-04-16T19:41:28Z

Never use forks ever?

yes, that's also a possibility. We could change our internal workflow, get the data somewhere, review them, and then someone uploads them directly on the master branch.

alexfoias · 2021-04-19T14:24:35Z

@kousu Thanks for fixing the issue. Is there an update in the documentation for how we should handle data in the future ?

jcohenadad · 2021-04-19T14:26:03Z

Should we consider dropping forks/branches, and just having someone from the team review the addition and push it directly on master?

kousu · 2021-11-29T16:46:36Z

We had a report by email ("git-annex at UMN servers") about these 51 files not being downloadable. I don't see how, CI has been green for the 8 months since they went in, but git annex is an arcane art and maybe the unusual fixup I did broke something somehow when combined with certain git annex versions.

alexfoias requested a review from jcohenadad March 24, 2021 22:15

alexfoias marked this pull request as ready for review March 25, 2021 13:16

jcohenadad reviewed Apr 5, 2021

View reviewed changes

jcohenadad approved these changes Apr 12, 2021

View reviewed changes

kousu mentioned this pull request Apr 12, 2021

git-annex in a separate branch neuropoly/data-management#67

Open

kousu mentioned this pull request Apr 13, 2021

git-annex leaves cruft remotes around neuropoly/data-management#71

Open

kousu mentioned this pull request Apr 16, 2021

Handle testing forks. #83

Merged

kousu force-pushed the af/add_ucdavis branch from 22ea4ae to e3571e2 Compare April 16, 2021 18:42

alexfoias added 3 commits April 16, 2021 14:58

add sub-ucdavis01, sub-ucdavis02, sub-ucdavis03,

8a08f34

sub-ucdavis04, sub-ucdavis05, sub-ucdavis06, sub-ucdavis07

update age and sex for ucdavis

191813f

fix participants.tsv

85af4c7

kousu force-pushed the af/add_ucdavis branch from e3571e2 to 85af4c7 Compare April 16, 2021 19:02

kousu merged commit 46df262 into spine-generic:master Apr 16, 2021

kousu mentioned this pull request Apr 16, 2021

Sync git-annex for #77 #84

Merged

kousu mentioned this pull request Apr 18, 2021

Adding height and weight into participants #85

Merged

alexfoias mentioned this pull request May 26, 2021

Substitute dash with n/a in the participants.tsv #89

Closed

jcohenadad added this to the next-release milestone Nov 10, 2021

add site data ucdavis #77

add site data ucdavis #77

Conversation

alexfoias commented Mar 24, 2021

alexfoias commented Mar 25, 2021

jcohenadad Apr 5, 2021

Choose a reason for hiding this comment

jcohenadad Apr 5, 2021

Choose a reason for hiding this comment

alexfoias Apr 12, 2021

Choose a reason for hiding this comment

jcohenadad Apr 5, 2021

Choose a reason for hiding this comment

alexfoias Apr 12, 2021

Choose a reason for hiding this comment

jcohenadad Apr 5, 2021

Choose a reason for hiding this comment

alexfoias Apr 12, 2021

Choose a reason for hiding this comment

alexfoias commented Apr 12, 2021

jcohenadad commented Apr 12, 2021

alexfoias commented Apr 12, 2021

kousu commented Apr 12, 2021

kousu commented Apr 12, 2021 • edited Loading

kousu commented Apr 12, 2021 • edited Loading

alexfoias commented Apr 13, 2021

kousu commented Apr 13, 2021

alexfoias commented Apr 13, 2021

kousu commented Apr 13, 2021 • edited Loading

kousu commented Apr 13, 2021

alexfoias commented Apr 16, 2021

kousu commented Apr 16, 2021

alexfoias commented Apr 16, 2021

alexfoias commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

alexfoias commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021

jcohenadad commented Apr 16, 2021

kousu commented Apr 16, 2021

kousu commented Apr 16, 2021 • edited Loading

kousu commented Apr 16, 2021

jcohenadad commented Apr 16, 2021

alexfoias commented Apr 19, 2021

jcohenadad commented Apr 19, 2021

kousu commented Nov 29, 2021

kousu commented Apr 12, 2021 •

edited

Loading

kousu commented Apr 12, 2021 •

edited

Loading

kousu commented Apr 13, 2021 •

edited

Loading

kousu commented Apr 16, 2021 •

edited

Loading