Enable cloning via Git Wire Protocol v2 over HTTP

[billiegoose]

This change makes it possible to clone Gitea repositories via HTTP(S) using Git Wire Protocol Version 2, which is not currently possible. Original discussion started with my comment here.

TL;DR - we need to run the serviceRPC and getInfoRefs with a new environment variable, GIT_PROTOCOL.

I compiled and ran it locally using the sqlite backend and successfully got partial clone working! (It requires running git config uploadpack.allowfilter 1 on each repository in ~/gitea-repositories, but partial clone is considered experimental so some extra work to enable it seems fair.)

[lunny]

The wrap function seems unnecessary?

[billiegoose]

Honestly, this file scares me lol. They all look like mostly unnecessary wrapper functions to me. But I was trying to go along with the style. 🤷‍♂️

[zeripath]

Just use RunInDirTimeoutEnv(env, -1, dir) directly instead. At some point we'll hopefully destroy all of these and replace them with a single RunOptions{} that can have things set.

Honestly, usually what happens is I end up escalating all the way up to FullPipeline to use Readers because a lot of functions misuse storing unbounded results and/or the error munging with these downstream functions ends up being unhelpful.

[willstott101]

Is there some central location where one could setup uploadpack.allowfilter 1? If we have to run it via SSH on all repos then new repos won't benefit from partial clone?

Even if it's experimental a central flag would be great

[zeripath]

@willstott101 is that a git config flag? You can just add it to the global or Gitea user .gitconfig

[billiegoose]

@willstott101 is that a git config flag? You can just add it to the global or Gitea user .gitconfig

🤔 Yeah.... that sounds like it should work... I haven't had success with that approach yet for some reason. Does gitea use a different ~/.gitconfig file? I thought it was running as my normal user.

[billiegoose]

So setting the value in ~/.gitconfig seems to work locally:

> git config --global uploadpack.allowfilter true
> GIT_PROTOCOL=version=2 git upload-pack --stateless-rpc --advertise-refs .
000eversion 2
0015agent=git/2.27.0
000cls-refs
0019fetch=shallow filter
0012server-option
0000
> git config --global uploadpack.allowfilter false
> GIT_PROTOCOL=version=2 git upload-pack --stateless-rpc --advertise-refs .
000eversion 2
0015agent=git/2.27.0
000cls-refs
0012fetch=shallow
0012server-option
0000

But when gitea runs git upload-pack it seems like it isn't picking up that setting, and always returns fetch=shallow and not fetch=shallow filter unless I specifically run git config uploadpack.allowfilter true in the repo. :/ Maybe it's a shell environment setting or something...

[billiegoose]

Ah. I figured out why ~/.gitconfig isn't respected. getInfoRefs runs without any of the host environment variables, crucially the $HOME environment variable. If I add:

		if protocol := h.r.Header.Get("Git-Protocol"); protocol != "" && safeGitProtocolHeader.MatchString(protocol) {
			h.environ = append(h.environ, "GIT_PROTOCOL="+protocol)
		}
+		h.environ = append(h.environ, "HOME="+os.Getenv("HOME"))

		refs, err := git.NewCommand(service, "--stateless-rpc", "--advertise-refs", ".").RunInDirTimeoutEnv(h.environ, -1, h.dir)

then it picks up that uploadpack.allowfilter is true and advertises fetch=shallow filter as desired.

I'm thinking the most reliable thing to do would be to run getInfoRefs with all the host environment variables like we do for serviceRPC:

cmd.Env = append(os.Environ(), h.environ...)

[zeripath]

Damn missed that you had done that... os.Environ() is used if you don't pass a different environment. You need to apply my suggestion

[billiegoose]

Woohoo! Thanks y'all!