Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support uploading over http(s):// #7

Closed
kousu opened this issue May 8, 2022 · 4 comments
Closed

Support uploading over http(s):// #7

kousu opened this issue May 8, 2022 · 4 comments

Comments

@kousu
Copy link
Member

kousu commented May 8, 2022 •
edited

This is a follow up to #3: filling in the blanks from https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/, I thought that only git annex get worked with http remotes. But now I think that was premature.

Investigate and make git-annex uploading work over http remotes.

Specifically this means these commands should work:

  • git annex sync --content
  • git annex copy --to origin
@kousu
Copy link
Member Author

kousu commented May 8, 2022 •
edited

Using a test repo at https://data.praxisinstitute.org.dev.neuropoly.org/uofc/CANDICE-fMRI-, which is currently running the patch for #3, download anonymously:

git clone ... && git annex get 
[kousu@nigiri down]$ git clone https://data.praxisinstitute.org.dev.neuropoly.org/UofC/CANDICE-fMRI-.git
Cloning into 'CANDICE-fMRI-'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 71 (delta 21), reused 0 (delta 0)
Receiving objects: 100% (71/71), 9.72 KiB | 4.86 MiB/s, done.
Resolving deltas: 100% (21/21), done.
[kousu@nigiri down]$ cd CANDICE-fMRI-/
[kousu@nigiri CANDICE-fMRI-]$ git annex get
get sub-BAN01/anat/sub-BAN01_T1w.nii.gz (from origin...) 
47%   2.87 MiB        895 KiB/s 3s    
ok                                    
get sub-BAN02/anat/sub-BAN02_T1w.nii.gz (from origin...) 
52%   3.31 MiB          2 MiB/s 1s
ok                                
get sub-BAN03/anat/sub-BAN03_T1w.nii.gz (from origin...) 
ok                                
(recording state in git...)

then construct an new annexed file:

cp -r sub-BAN02/ sub-BAN04
dd if=/dev/urandom of=sub-BAN04/anat/sub-BAN02_T1w.nii.gz count=512
git add sub-BAN04/
git commit -m "Subject 04"

prove it's distinct (it's the last one, the one with 1 copy):

git annex whereis 
[kousu@nigiri CANDICE-fMRI-]$ git annex whereis
whereis sub-BAN01/anat/sub-BAN01_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN02/anat/sub-BAN02_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN03/anat/sub-BAN03_T1w.nii.gz (2 copies) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
        d57145f0-6115-47c4-8278-994da90074c1 -- gitea@data.praxisinstitute.org.dev.neuropoly.org:~/data/gitea-repositories/muhc/candice-fmri-.git [origin]
ok
whereis sub-BAN04/anat/sub-BAN02_T1w.nii.gz (1 copy) 
        d4d1a0b0-aa26-421f-b1bb-43adf5e9a6cb -- kousu@nigiri:~/src/neuropoly/neurogitea/down/CANDICE-fMRI- [here]
ok

and now see what happens if we try to upload it; with debugging turned on:

git annex copy --to origin 
[kousu@nigiri CANDICE-fMRI-]$ git config annex.debug true
[kousu@nigiri CANDICE-fMRI-]$ git annex copy --to origin
[2022-05-08 02:52:23.217458919] (Utility.Process) process [28036] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2022-05-08 02:52:23.219000083] (Utility.Process) process [28036] done ExitSuccess
[2022-05-08 02:52:23.219516464] (Utility.Process) process [28037] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2022-05-08 02:52:23.221135085] (Utility.Process) process [28037] done ExitSuccess
[2022-05-08 02:52:23.22179462] (Utility.Process) process [28038] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..6c3b23ed5b89e73fdb170e4dfb3dc4f7324acd87","--pretty=%H","-n1"]
[2022-05-08 02:52:23.224259745] (Utility.Process) process [28038] done ExitSuccess
[2022-05-08 02:52:23.225976282] (Utility.Process) process [28039] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2022-05-08 02:52:23.228332152] (Utility.Process) process [28040] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--stage","-z","--error-unmatch","--"]
[2022-05-08 02:52:23.229159541] (Utility.Process) process [28041] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch-check=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2022-05-08 02:52:23.22958377] (Utility.Process) process [28042] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
[2022-05-08 02:52:23.230213916] (Utility.Process) process [28039] done ExitSuccess
[2022-05-08 02:52:23.231313697] (Utility.Process) process [28043] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch=%(objectname) %(objecttype) %(objectsize)","--buffer"]
copy sub-BAN01/anat/sub-BAN01_T1w.nii.gz [2022-05-08 02:52:23.269482287] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/c34/98e/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:23.715269089] (Utility.Process) process [28045] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:33.439560188] (Utility.Process) process [28045] done ExitSuccess
[2022-05-08 02:52:33.439918575] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/c34/98e/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:33.531585766] (Utility.Process) process [28081] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:33.533291906] (Utility.Process) process [28081] done ExitSuccess
[2022-05-08 02:52:33.53378081] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/73/34/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:33.622360074] (Utility.Process) process [28082] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:38.54396353] (Utility.Process) process [28082] done ExitSuccess
[2022-05-08 02:52:38.544127686] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/73/34/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz/SHA256E-s6472001--453e19abdba0eaa546a2dcb3ff72f71b1540fe34b5ce9aa0df841cf00c790dc2.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:38.629070613] (Utility.Process) process [28086] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:38.630412108] (Utility.Process) process [28086] done ExitSuccess
(not found) failed
copy sub-BAN02/anat/sub-BAN02_T1w.nii.gz [2022-05-08 02:52:38.63124213] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:38.717445234] (Utility.Process) process [28087] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:43.084445916] (Utility.Process) process [28087] done ExitSuccess
[2022-05-08 02:52:43.084688615] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:43.170468809] (Utility.Process) process [28088] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:43.172434906] (Utility.Process) process [28088] done ExitSuccess
[2022-05-08 02:52:43.172943361] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:43.258412042] (Utility.Process) process [28089] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:46.248660233] (Utility.Process) process [28089] done ExitSuccess
[2022-05-08 02:52:46.249043634] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:46.335692255] (Utility.Process) process [28090] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:46.337206776] (Utility.Process) process [28090] done ExitSuccess
(not found) failed
copy sub-BAN03/anat/sub-BAN03_T1w.nii.gz [2022-05-08 02:52:46.337740833] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/58d/8ea/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:46.423388602] (Utility.Process) process [28091] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:49.262936219] (Utility.Process) process [28091] done ExitSuccess
[2022-05-08 02:52:49.263330345] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/58d/8ea/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:49.349186911] (Utility.Process) process [28092] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:49.350967438] (Utility.Process) process [28092] done ExitSuccess
[2022-05-08 02:52:49.35138871] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/1X/Mx/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:49.435139518] (Utility.Process) process [28093] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:52.986881594] (Utility.Process) process [28093] done ExitSuccess
[2022-05-08 02:52:52.987023514] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/1X/Mx/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz/SHA256E-s5688327--c9c8ad310aa906d9aea87f251932f7adba3120fd39f6a621c27b531c6ecfcce9.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:53.076025925] (Utility.Process) process [28096] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:53.077770146] (Utility.Process) process [28096] done ExitSuccess
(not found) failed
copy sub-BAN04/anat/sub-BAN02_T1w.nii.gz [2022-05-08 02:52:53.078448893] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:53.16577961] (Utility.Process) process [28097] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:52:56.868778349] (Utility.Process) process [28097] done ExitSuccess
[2022-05-08 02:52:56.869119387] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/433/2b9/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:56.954725712] (Utility.Process) process [28098] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:52:56.956656836] (Utility.Process) process [28098] done ExitSuccess
[2022-05-08 02:52:56.957140664] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:52:57.041148655] (Utility.Process) process [28099] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","fill"]
Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': kousu
Password for 'https://kousu@data.praxisinstitute.org.dev.neuropoly.org': 
[2022-05-08 02:53:00.095495453] (Utility.Process) process [28099] done ExitSuccess
[2022-05-08 02:53:00.095774849] (Utility.Url) Request {
  host                 = "data.praxisinstitute.org.dev.neuropoly.org"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Authorization","<REDACTED>"),("User-Agent","git-annex/10.20220322-g959beeea9")]
  path                 = "/UofC/CANDICE-fMRI-.git/annex/objects/93/kp/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz/SHA256E-s6677912--29be5e1eef0d3b5fcd6817999c9055f6c088d58b37c7c09bb87c440fb8037c81.nii.gz"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[2022-05-08 02:53:00.183376947] (Utility.Process) process [28100] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","credential","reject"]
[2022-05-08 02:53:00.18501078] (Utility.Process) process [28100] done ExitSuccess
(not found) failed
[2022-05-08 02:53:00.185253039] (Utility.Process) process [28043] done ExitSuccess
[2022-05-08 02:53:00.185356286] (Utility.Process) process [28042] done ExitSuccess
[2022-05-08 02:53:00.185460462] (Utility.Process) process [28041] done ExitSuccess
[2022-05-08 02:53:00.185552806] (Utility.Process) process [28040] done ExitSuccess
copy: 4 failed

Interestingly, git-annex is sending an Authorization header, and it looks like it's leaning on git credential so so it's git doing the prompting, which is nice and elegant. So maybe uploading over HTTP does work afterall.

But this part of the protocol isn't documented on the wiki, or at least I don't want to waste my time trying to find my way around that place. But I bet we have enough here to reverse engineer it; it looks like the first thing it needs is a response to a HEAD request, presumably checking if the file already exists or not; so we'll have to add another route for that, and once we do that maybe it'll send a POST or a PUT request to actually upload the file, and we'll have to add a route for that too. And then it should be done 🤞

@kousu kousu mentioned this issue May 8, 2022
Merged
@kousu
Copy link
Member Author

kousu commented Sep 20, 2022 •
edited

I tried again to get this working for a few hours but no matter what I tried I couldn't convince git-annex to do it.

I learnt it issues a HEAD request for each file. If that returns 200, and the Content-Length matches the expected size, it says "ok". Otherwise, it says "not found". It never tries to issue a PUT or a POST. So I doubt this is supported.

I suspect it's a result of there being no server in the world that it could possibly be tested against. GitHub doesn't support it, Apache doesn't support PUTs, GitLab probably doesn't either. If we added annex HTTP uploading support we'd be the first, probably.

I'll put in a feature request and see what joeyh says.

@kousu kousu changed the title Support git annex sync --content/git annex copy --to over http(s):// Support uploading over http(s):// Sep 20, 2022
@kousu
Copy link
Member Author

kousu commented Sep 20, 2022

Feature request/request for clarification: https://git-annex.branchable.com/forum/HTTP_uploads/?updated

@kousu
Copy link
Member Author

kousu commented Sep 23, 2022

This is a wontfix:

There is no "one true way" to upload a file to a HTTP server.

I don't want to add some additional git-annex specific HTTP-based protocol.

(I lightly disagree: the One True Way is PUT + an Authorization: header, which is just what S3 uses, and what Gitea could easily support. But if the code doesn't exist it doesn't exist.)

@kousu kousu closed this as not planned Won't fix, can't repro, duplicate, stale Sep 23, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 9, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kousu