Support downloading from "dumb http" with git-annex.

[kousu]

This makes git clone https://...... && git annex get function, so that datasets can be shared widely, without the big roadblock of necessarily having to create accounts or ssh keys for users.

Fixes #3

I don't think git-annex supports uploading over http, and
neither does this. If it turns out it does I'll revise this.

Ref: https://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/
but this skip several of those instructions:

• they assume you're deploying to a non-bare repository
• gitea already takes care of the update-server-info hook
• we want to enforce gitea's access control so we don't
  use core.sharedRepository

Follows #18

[kousu]

Some things to test:

• what happens if you try uploading anyway? Does git annex copy --to just fail without even trying to contact the server? (-> see Support uploading over http(s)://  #7 (comment))
• does git-annex prompt for a username/password when used against a private repo over https?
• does DroneCI work? (see: https://github.com/neuropoly/computers/tree/master/ansible/roles/neuropoly.droneci (sorry, not yet public))? Can it get and test annexed files?
• that access control behaves itself
  • private repos should not be readable
  • a repo that was public and then becomes private should not allow git annex get to get files added after it became private; and/or it should prompt for a password
• does it tolerate case-insensitive repo URLs? (-> Support downloading from "dumb http" with git-annex. #6 (comment))
• does GIN support https clones?
  Yes, people are using it (and having bugs, but getting around them):
  • https://git-annex.branchable.com/bugs/GIN_is_not_recognized_as_a_git-annex_enabled_repos/
  • https://git-annex.branchable.com/bugs/be_like_git_and_ask_for_credentials_if_404/
  • https://git-annex.branchable.com/bugs/leaks_git_config_error_message_upon_inability_to_read_downloaded___34__config__34___file/

Some things to nitpick:

• my comments are too wordy

[kousu]

There's a build of this at https://github.com/neuropoly/gitea/releases/tag/v1.16.7-git-annex-http (rebased on v1.16.7 instead of main, to make it safer to deploy). I've installed it on our current dev server https://data.praxisinstitute.org.dev.neuropoly.org/.

Deploying it made this work on the pre-existing annexed dataset we had there:

[kousu@nigiri down]$ git clone https://data.praxisinstitute.org.dev.neuropoly.org/uofc/candice-fmri-.git
Cloning into 'candice-fmri-'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 71 (delta 21), reused 0 (delta 0)
Receiving objects: 100% (71/71), 9.72 KiB | 9.72 MiB/s, done.
Resolving deltas: 100% (21/21), done.
[kousu@nigiri down]$ cd candice-fmri-/
[kousu@nigiri candice-fmri-]$ git annex get
get sub-BAN01/anat/sub-BAN01_T1w.nii.gz (from origin...) 
ok                                   
get sub-BAN02/anat/sub-BAN02_T1w.nii.gz (from origin...) 
ok                                
get sub-BAN03/anat/sub-BAN03_T1w.nii.gz (from origin...) 
ok                                
(recording state in git...)

[kousu]

I set that repo back to its mixed-case name:

and that worked too:

[kousu@nigiri down]$ git clone https://data.praxisinstitute.org.dev.neuropoly.org/UofC/CANDICE-fMRI-.git
Cloning into 'CANDICE-fMRI-'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 71 (delta 21), reused 0 (delta 0)
Receiving objects: 100% (71/71), 9.72 KiB | 4.86 MiB/s, done.
Resolving deltas: 100% (21/21), done.
[kousu@nigiri down]$ cd CANDICE-fMRI-/
[kousu@nigiri CANDICE-fMRI-]$ git annex get
get sub-BAN01/anat/sub-BAN01_T1w.nii.gz (from origin...) 
47%   2.87 MiB        895 KiB/s 3s    
ok                                    
get sub-BAN02/anat/sub-BAN02_T1w.nii.gz (from origin...) 
52%   3.31 MiB          2 MiB/s 1s
ok                                
get sub-BAN03/anat/sub-BAN03_T1w.nii.gz (from origin...) 
ok                                
(recording state in git...)

so it looks like case-sensitivity isn't a problem.

[mguaypaq]

It's great that you got this working!!! Here are my comments on the code as it stands. You can ping me again if/when you want another review.

[kousu]

A test against Drone: https://drone.data.praxisinstitute.org.dev.neuropoly.org/nguenthe/CANDICE-fMRI-/2/1/3

It did not work:

+ git fetch --depth=1 origin git-annex:git-annex
From https://data.praxisinstitute.org.dev.neuropoly.org/nguenthe/CANDICE-fMRI-
 * [new branch]      git-annex  -> git-annex
 * [new branch]      git-annex  -> origin/git-annex
+ git annex get -J8
(scanning for unlocked files...)
fatal: could not read Username for 'https://data.praxisinstitute.org.dev.neuropoly.org': terminal prompts disabled

  Remote origin not usable by git-annex; setting annex-ignore
get sub-BAN01/anat/sub-BAN01_T1w.nii.gz (not available) 
[...]

"terminal prompts disabled" is the sort of problem you get when stdin is closed or not a tty. But this is a private repo:

and the clone clearly worked, and even the git fetch worked, so I'll have to look into why that doesn't extend to git annex get.

Solving this is not "on the critical path", because we're planning to drop DroneCI anyway and use @mguaypaq's localhost bids-validator Gitea plugin, and our main use case for HTTP downloads is for sharing open access data, but I'd really like to not leave this half-done if I can.