Improve installation documentation for Windows users

[ViliusS]

Since Windows do allow to install Python more easily these days and takes care of most PATH stuff, use recommendations from Microsoft.

The only hiccup is that Git for Windows cannot parse get-filter-repo SHEBANG value when the script is installed via pip. That value is very long by default. Changing it to #!\python.exe solves the issue.

Tested under clean installation of Windows 10 Enterprise version 20H2.

[newren]

Is this obsolete with @dscho's changes to Git for Windows to make it accept the Windows Store version of python? (See comments in #231.)

[ViliusS]

Sadly no. I still cannot run git filter-repo without changing shebang value to #!\python.exe or #!/usr/bin/python under latest Git for Windows 2.31.1.

I can run it via python git-filter-repo though, so it must be something in Git for Windows indeed.

I'm using Windows Command Promt as Git for Windows shell by the way. @dscho's fix is probably for Git Bash only?

[dscho]

Does it work for you if you execute the command in Git Bash?

[dscho]

Also, does python3 ... work?

[ViliusS]

Nope.

username@PC MINGW64 ~
$ git filter-repo
error: cannot spawn git-filter-repo: Exec format error

To summarize, these shebang values work:
#!\python3.9.exe
#!\python3.exe
#!\python.exe (all of these do work with and without .exe extension which is standard under CMD)
#!/python (which is probably the same as \python)
#!/usr/bin/python
#!/usr/bin/env python (last two probably uses msys path translation to resolve to python.exe)

And these do not work:
#!C:\Users\username\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe (<-- default when installed with pip from MS Store)
#!C:\Users\username\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python3.exe
#!/C/Users/username/AppData/Local/Microsoft/WindowsApps/PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0/python.exe

Maybe it's hitting path limit or something?

UPDATE: I just did a simple test. Created symbolic link: ``` C:\>mklink /d \test C:\Users\username\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0 symbolic link created for \test <<===>> C:\Users\username\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0 ```

UPDATE2: even easier way to check, no need to create symlink. Just use something like:
#!asdadsasdasdddddddddddddddddssssssssssssssssssssssssssssddddddddddddddddddddddddddddd\python.exe (this works)
#!asdadsasdasdddddddddddddddddssssssssssssssssssssssssssssdddddddddddddddddddddddddddddd\python.exe (this doesn't)
#!asdadsasdasdddddddddddddddddssssssssssssssssssssssssssssddddddddddddddddddddddddddddddddd\python (works)
#!asdadsasdasdddddddddddddddddssssssssssssssssssssssssssssdddddddddddddddddddddddddddddddddd\python (doesn't work)

There is definitely something wrong with that path size in msysgit path conversion.

[dscho]

It's not msysgit.

Anyway, what do where python and where python3 say in CMD?

[dscho]

And: do python -h and python3 -h work in your Git Bash?

[ViliusS]

> where python
C:\Users\username\AppData\Local\Microsoft\WindowsApps\python.exe

(which is a reparse point and points to C:\Users\username\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe same for python3)

python -h, using python interactively and python git-filter-repo works so that's not an issue with PATHs or permissions. Shebang value has 96 character limit when running from Git for Windows for some reason.

[dscho]

Maybe this is the limit right there: https://github.com/git-for-windows/git/blob/8b8cd53ead5e816b846ab5d26052afa3f4d62c13/compat/mingw.c#L1484

[ViliusS]

I don't know enough about C to understand why the real limit is 96 and not 100, but yea, it could be it.

[dscho]

What I really don't understand is that the hash-bang line should read #!/usr/bin/env python3, i.e. it should resolve the interpreter to env.exe (which is in C:\Program Files\Git\usr\bin\env.exe, a path well below that limit).

BTW I broke down and interrupted my AFK to play with this a bit, and I can finally reproduce your Exec format error. I think it will be better to fix this on Git for Windows' side than on git-filter-repo's. Hopefully I'll find some time to play more with that, for now I have to go back to AFK.

In the meantime, could I ask you to figure out an easy way to add the script location to the PATH in ~/.bashrc? That would make much more sense than to copy git-filter-repo into Git's exec-path, as the latter method is prone to let the copied version go stale upon git-filter-repo upgrades.

[ViliusS]

Shebang value is autogenerated by the packager which is used to install git-filter-repo which in my case is pip from the MS Store. I'm afraid we won't have control over it.

I do agree that path limit needs to be fixed in Git for Windows, however git-filter-repo documentation still needs to be improved and fixing that limit only removes step 4 in my instructions, so this pull request still stands.

Regarding adding Scripts folder to PATH, I thought about it for some time but could not find an easy way make it fully automated. If we copy git-filter-repo to Git's exec-path it won't be upgraded by the next pip upgrade git-filter-repo and if we add that folder to Windows PATH it does contain hardcoded Python version which could break on next Python upgrade, e.g. 3.9 -> 3.10.

Adding Scripts folder to .bashrc still leaves us with broken path with next Python upgrade and only works when running Git for Windows with Bash, not cmd. I believe the correct way would be to contact Python Software Foundation to see if they can implement this during MS Store installation, however I didn't find an easy way to leave this feedback for them.

[dscho]

Maybe printf '#!/bin/sh\n\nexec python git_filter_repo "$@"\n' >~/bin/git-filter-repo would still be the best solution.

[ViliusS]

I doubt that these instructions are easier than to copy to Git's exec path :) But maybe it's just me.

[newren]

So...

I'm not particularly happy with the really lengthy instructions. They suggest there is a more fundamental issue that needs to be solved elsewhere, and since I have multiple reports of people using filter-repo successfully on Windows, I'm worried the instructions might break some of those users while fixing it for others. If it were instead another issue or something I could link to that people could check, or if we could root cause that it really affected everyone, then I'd be more at ease that these instructions wouldn't break others.

But fundamentally, there are a few more issues with these instructions:

1. The suggestion that installation should be done via pip (as opposed to could be done) is out of place. The very first people to package git-filter-repo for Windows and submit things to fix up git-filter-repo were the Scoop folks. They were faster than nearly all the linux packaging folks, in fact. And I've never heard a complaint from anyone installing via the Scoop package manager on Windows that things didn't just work. Yet I've heard multiple complaints from folks using pip, making me nearly regret even suggesting pip as a possibility. I moved it from the top of the page (as put there by the folks who did the pip packaging to make it possible) to near the bottom already, maybe I need to bury it even more.

2. I don't think explaining how to install dependencies belongs in the scope of filter-repo. (The Python project should provide instructions on how to install python; not me. Also, if I'm adding instructions on how to install python, on any platform, I probably need to start adding it for other platforms. Those will probably need lots of updates over time and I'm putting myself into a losing game). So steps 1 & 2 do not belong here.

3. The instructions suggest pip might be buggy. If so, why is it even being suggested? In particular, it suggests both that pip isn't sticking the script in $PATH (in which case is that Windows-specific or all platforms? If all platforms, it doesn't belong in the windows specific section. It looks like I already noted that problem under pip, so repeating it in the windows specific section seems duplicative, so step 3 really doesn't fit either). The instructions also suggest that the shebang mangling pip is doing might be incorrect (though Johannes suggests the shebang handling might be a Git for Windows bug rather than a pip one).

I think all that's left after the above is (a) a fix for the shebang handling in Git for Windows, and (b) maybe being more overt on the HERE-BE-DRAGONS warning in the pip section about how it can cause problems for old versions of Git for Windows (anything released prior to April 2021), and additionally that it leaves to the users one of the installation steps -- copying the script into $PATH -- and that perhaps folks should strongly consider using other package managers instead.

[dscho]

Maybe this is the limit right there: https://github.com/git-for-windows/git/blob/8b8cd53ead5e816b846ab5d26052afa3f4d62c13/compat/mingw.c#L1484

I got a chance to try out this hypothesis. Increasing the size from 100 to 248 fixes the Exec format error bug.

[dscho]

Increasing the size from 100 to 248 fixes the Exec format error bug.

I opened git-for-windows/git#3165 to address that.

[newren]

Increasing the size from 100 to 248 fixes the Exec format error bug.

I opened git-for-windows/git#3165 to address that.

Awesome, thanks!

[ViliusS]

So...

I'm not particularly happy with the really lengthy instructions. They suggest there is a more fundamental issue that needs to be solved elsewhere, and since I have multiple reports of people using filter-repo successfully on Windows, I'm worried the instructions might break some of those users while fixing it for others. If it were instead another issue or something I could link to that people could check, or if we could root cause that it really affected everyone, then I'd be more at ease that these instructions wouldn't break others.

I will answer your other points in a minute, but let me first explain my experience as a git-filter-repo user.

Current note for Windows users in install docs suggests that there is something wrong with "default python" on Windows. OK, so I went to Python.org and tried to install it from there. I have also tried to download git-filter-repo from this repository and just place the file in my PATH as suggested. This didn't work. I have searched several rabbit holes coming empty handed. Since there are multiple ways how Python can be installed on Windows from the linked issues I had to guess if that relates to my setup or not (most of the time it didn't). After digging for several hours, to the point where I forgot why do I need Python anyway, I went back to using git filter-branch.

I would never came back if not for gitlab.com which supports rewriting history only with git-filter-repo. This time I thought that I won't follow git-filter-repo documentation but just install everything as I always do, using standard Windows tools. First I had to find out that, even if documentation suggest so, there is no default python on Windows. OK, so I thought, let's try to install it. What is the best way? Naturally I consulted my OS documentation which pointed to MS Store or WSL installs. MS Store install was a breeze and I have confirmed that python was placed in my PATH via python --version etc. Documentation specifically states that Python from MS Store comes with pip which is the default package manager for Python, so I have used it to install git-filter-repo. Now running git filter-repo still could not find it but I have remembered this line git-filter-repo is a single-file python script, which was done to make installation for basic use trivial: just copy it into your $PATH. So I dig where pip put the file and added it to my PATH. With the exception of shebang hiccup, everything worked and it only took me 10 minutes instead of several hours when following documentation and reading through various issues.

Naturally, I was frustrated because of this and thought that nobody should go through the same experience as me, hence this PR.

But fundamentally, there are a few more issues with these instructions:

1. The suggestion that installation should be done via pip (as opposed to could be done) is out of place. The very first people to package git-filter-repo for Windows and submit things to fix up git-filter-repo were the Scoop folks. They were faster than nearly all the linux packaging folks, in fact. And I've never heard a complaint from anyone installing via the Scoop package manager on Windows that things didn't just work. Yet I've heard multiple complaints from folks using pip, making me nearly regret even suggesting pip as a possibility. I moved it from the top of the page (as put there by the folks who did the pip packaging to make it possible) to near the bottom already, maybe I need to bury it even more.

I can change it from "should" to "could" and your point about Scoop folks is probably correct, but I'm thinking this way: software under every OS should be installed via standard way as much as possible, be it rpm/deb on Linux, dmg on macOS, or MS Store/MSI on Windows. This way it automatically takes cares of OS specific issues, most of the time is auto-updated, provisioned, etc. etc. Not every developer uses Python for their work, I would argue that a fair amount of them doesn't even know what Python or pip is. Also, if you are HTML developer, or PHP developer, or C# developer on Visual Studio, or database developer, you probably won't have any of the package managers installed (I didn't). Requiring these folks to install separate package manager just to install one simple python script to be able to rewrite git history is insane.

If you are afraid that these instructions will break something, you can always revert them (using git-filter-repo to remove them from existence :)), but facts show that folks are still stuggling without them, jumping from one install method to another without any good reason, #193 #189 #124 Some of the issues they discovered are already fixed in latest Windows/Git for Windows versions, others are just not understanding how various Python install alternatives work and which PATHs are already defined and should not be touched. By the way most of these links suggest that installing from MS Store finally solved their problems.

2. I don't think explaining how to install dependencies belongs in the scope of filter-repo. (The Python project should provide instructions on how to install python; not me. Also, if I'm adding instructions on how to install python, on any platform, I probably need to start adding it for other platforms. Those will probably need lots of updates over time and I'm putting myself into a losing game). So steps 1 & 2 do not belong here.

I still think one or two lines how to install dependencies won't hurt and will solve more issues than it produces. Instructions are pointing to official MS' documentation and if something changes they will take care of it. Even if MS Store method won't be recommended some time in the future others (or me) could make another PR to update documentation, that's natural cycle of open source software. The same goes for other OSes, if anybody is willing to provide a PR to make more detailed instructions, why not?

3. The instructions suggest pip might be buggy. If so, why is it even being suggested? In particular, it suggests both that pip isn't sticking the script in $PATH (in which case is that Windows-specific or all platforms? If all platforms, it doesn't belong in the windows specific section. It looks like I already noted that problem under pip, so repeating it in the windows specific section seems duplicative, so step 3 really doesn't fit either). The instructions also suggest that the shebang mangling pip is doing might be incorrect (though Johannes suggests the shebang handling might be a Git for Windows bug rather than a pip one).

That's not correct. pip is not buggy, Python's install method doesn't register Scripts in the PATH, but I'm sure that will be solved sooner rather then later. I just need to find a way to reach Python folks to leave this feedback for them.

I think all that's left after the above is (a) a fix for the shebang handling in Git for Windows, and (b) maybe being more overt on the HERE-BE-DRAGONS warning in the pip section about how it can cause problems for old versions of Git for Windows (anything released prior to April 2021), and additionally that it leaves to the users one of the installation steps -- copying the script into $PATH -- and that perhaps folks should strongly consider using other package managers instead.

I will test it when the next Git for Windows version is released, share my experience and update the instructions if that's OK?