Up For Grabs: make installing the Git for Windows SDK as easy as cloning a repository

[dscho]

Currently, the Git for Windows SDK needs to be installed by running a self-extracting 7-Zip archive that eventually runs a .bat script to initialize our custom fork of MSYS2.

This is a bit tedious, as many things can go wrong on the way:

• there may be a proxy and pacman has to know about it
• pacman may want to update itself, which is a bit of a chicken-and-egg problem
• MSYS2's pacman repository (which really is a static website at https://repo.msys2.org) needs to be up and running
• Git for Windows' pacman repository (hosted by BinTray) needs to be up and running
• oh so many other things need to run smoothly

And even then, it is hard to assess how far along we are in the process.

The proposed solution

There is a silver lining on the horizon, though, as it should be a lot easier to install the SDK than this involved and fragile process

git-sdk-64.git and git-sdk-32.git

For the Continuous Testing of https://github.com/git/git, we already needed a more robust way to initialize and update Git for Windows SDKs. Imitating the way Git for Windows 1.x worked (whose SDK was called "msysgit", mistaken by many to be the end-user facing software), we have two repositories tracking the 64-bit and the 32-bit SDKs, respectively: https://github.com/git-for-windows/git-sdk-64 and https://github.com/git-for-windows/git-sdk-32

A trusty VSTS job keeps them up-to-date, at least once every 24h cycle, sometimes triggered manually (e.g. after updating a dependency).

The missing parts

These repositories already have most of what is needed to initialize a full Git for Windows SDK, but not quite:

• /usr/src/build-extra needs to be cloned,
• /usr/src/git needs to be cloned (and optionally built & installed into the SDK)
• a Desktop shortcut has to be created

Outline: the new SDK "installer"

So while it is totally possible to clone https://github.com/git-for-windows/git-sdk-64 and then double-click git-bash.exe, this will only open a Git Bash but leave a couple of things missing. The way we could make sure that those things are not missing is by detecting their absence and installing them in a script at startup.

Such a script already exists: /etc/profile.d/git-sdk.sh

This file is present only in Git for Windows SDKs (installed via the git-extra package), and it is excluded from the Git for Windows installer, portable Git, MinGit, etc

So the idea is simple: extend that script to install the missing parts, namely

• /usr/src/build-extra
• /usr/src/git (and build and install Git, unless it was already cloned)
• the Desktop shortcut

As with the current SDK installer, these steps should be guarded behind an test -z "$JENKINS_URL" || { ... }, to still allow installing the SDK into a Continuous Integration system where we may not want build-extra, git, and certainly not the shortcut.

It is also conceivable to introduce a new environment variable, say, GIT_SDK_OPTIONS with space-separated options to turn those features off on demand.

Document how to "install" the SDK in a README.md file

So far, the git-sdk-64 and git-sdk-32 repositories do not have a README.md file, which probably leaves occasional visitors wondering what those repositories are all about.

These files would make for an excellent location to describe what those repositories represent, how they are to be used, how they can be extended, how they are maintained, etc

[dscho]

This is now actually more urgent a project. It would be really nice if a volunteer could step up to do the job, as it is a pretty fun project, with next to no prior knowledge required (I will provide guidance, of course), and I personally do have a lot more non-fun projects to take care of in the meantime...

[bbolli]

I'm working on a series to implement this.

It is also conceivable to introduce a new environment variable, say, GIT_SDK_OPTIONS with space-separated options to turn those features off on demand.

The problem I have with this: when would a developer who just cloned the SDK repo set this variable? git-sdk.sh runs on first start of the shell and they get no opportunity to change the environment first. Maybe add a separate script that does the setup, that could have command line options?

[bbolli]

Next question: are any proxy settings (especially https_proxy) active when git-sdk.sh runs? Otherwise the git clones would fail if a proxy is needed to reach GitHub.

[bbolli]

Work in progress: https://github.com/bbolli/build-extra/commits/master

[dscho]

I'm working on a series to implement this.

Great!

The problem I have with this: when would a developer who just cloned the SDK repo set this variable? git-sdk.sh runs on first start of the shell and they get no opportunity to change the environment first. Maybe add a separate script that does the setup, that could have command line options?

True. Probably a Powershell script would make most sense, as that does not yet need the Bash, and therefore could even do things like set up the POSIX mqueue and stuff.

OTOH I could also imagine that the best course of action would be to actually not clone anything, but just initialize the repositories and set up remotes (not fetching anything yet).

And then we could introduce a shell function (maybe called sdk?) to help with common tasks, e.g.

• sdk build git
• sdk build installer
• sdk build msys2-runtime

The code would be very light-weight and mostly hand off to /usr/src/build-extra/please.sh, of course, after making sure that we have that file (fetching with --depth 1 if necessary, and checking out the files if necessary).

Next question: are any proxy settings (especially https_proxy) active when git-sdk.sh runs? Otherwise the git clones would fail if a proxy is needed to reach GitHub.

Good point. My thinking was to reuse the proxy helper introduced in git update-git-for-windows to determine what proxy to use: https://github.com/git-for-windows/build-extra/blob/42868e7be40e8f6cf61d18b26e97377833db8d04/git-extra/git-update-git-for-windows#L141-L155

(In this instance, I could imagine that we would look at config setting http.proxy, fall back to the https_proxy variable, and if it does not exist, query the proxy via the helper, and then set http.proxy, or if it is not necessary record that fact via git config --system http.proxy.notRequired true so that we do not go through the same shebang next time the SDK starts up.)

Work in progress: https://github.com/bbolli/build-extra/commits/master

Nice!

How about opening a [DO-NOT-MERGE] Pull Request already, to make reviewing easier? We can always edit out that [DO-NOT-MERGE] from the PR title when it is ready...