Determine the msys/python environment requirements for Servo on Windows

[larsbergstrom]

Related to #1908, in order to get Servo working on Windows we need to figure out the python installations and MSYS-related requirements for the build environment to work.

We have tried a couple of things:

1. Install both the native and MSYS versions of python. Patch the native installation to use directories expected by the MSYS version (e.g., bin instead of Scripts inside of virtualenv):
   https://hackpad.com/Servo-on-Windows-C1LPcI2bP25
   Builds happen within an MSYS bash shell.

2. Use a native python install, with msys tools installed via git-scm (http://git-scm.com/download/win), just for the binaries (i.e., not the shell is powershell or cmd.exe, not msys-bash).

In both cases, we are having problems related to pip installs during bootstrapping. In the first case, everything seems to download and get installed, but the packages are then not available in subsequent builds, seemingly because the installation is not finalized. Suggestions have been that the patches are incomplete and might require additional changes to pip and some of the site-related python code.

In the second case, packages end up both locally and globally installed by pip (that is, pip list from a shell shows them globally installed, not just to the virtualenv). But, we still get errors about a missing toml package.

WIP is available at https://github.com/larsbergstrom/servo/tree/win32

CC @vvuk, @Jayflux

[vvuk]

1. Install both the native and MSYS versions of python.

To avoid confusion, there's actually three possible versions of python --

1. The native win32 python. This comes from python.org. In this python, sys.platform == "win32", os.path.sep == "", os.name == "nt"
2. A MSYS2 python. This is the "msys2/python2" package, and is installed in /usr/bin/python. In this python, sys.platform == "msys", os.path.sep == "/", os.name == "posix"
3. A mingw64 python. This is the "mingw64/mingw-w64-x86_64-python2" package, installed in /mingw64/bin/python. In this python, sys.platform == "win32", os.path.sep == "/", os.name == "nt"

I ignored the MSYS2 python entirely, but maybe that was a mistake. That python uses the msys posix emulation layer, at might be the closest thing that might "just work". To get pip etc there, you have to install the "msys2/python-setuptools" package, then use "/usr/bin/easy_install-2.7 pip virtualenv".

If I do that, and then run "./mach build -d" from a MSYS2 shell (not MINGW64!), and run it as Administrator (required so that os.symlink() can work, otherwise virtualenv errors out there).... things progress to downloading a rustc nightly!

[vvuk]

mingw-w64-x86_64-python2 is required by mingw-w64-x86_64-gdb, so it can't easily be removed (without also removing gdb), but you can rename /mingw64/bin/python2.exe and python2.7.exe to python2-mingw64.exe etc. Now if I run ./mach build -d from within a mingw64 shell, I'm downloading the proper rust nightly. (This was easily fixable in a msys shell too, but I suspect that this is going to result in more happiness, since gcc etc in the mingw64 env is more what we want)

[vvuk]

So, build seems to be going now. What I did, from a MINGW64 shell started as Administrator (right click, run as Administrator; or check run as Administrator in something like ConEmu).

cd /mingw64/bin
mv python2.exe python2-mingw64.exe
mv python2.7.exe python2.7-mingw64.exe
pacman -S python2-setuptools
easy_install-2.7 pip virtualenv
./mach build -d

[jasonwilliams]

@vvuk is there a better fix for this long term?

[vvuk]

No idea. The better fix is to not use python in the build system (and especially not use virtualenv etc.), but that ship has sailed.

[zwn]

I am slowly getting something to build (see #10596 and #10595). The steps I used:

1. install python (original win32)
2. install msys2
3. somehow make 'mach' to create virtualenv using the win32 python
   • one way to do it is to open mingw64 shell and put python install path at the front of PATH
   • other way is to create the _virtualenv explicitly using the win32 python
4. fix #10595
5. get stuck at #10596

I was not aware there are two different pythons in msys2. I have tried the mingw64 version, but virtualenv is not working there. I will try the msys version to see if it makes any difference for #10596.

[zwn]

Well, it fails even sooner while "Compiling hbs-pow-sys v0.2.0" saying virtualenv is not working (it hardcodes win32 python).

[zeusintuivo]

I have researched and documented an effective way to have windows environments set up for 32 and 64 bits in Windows 7. I assume this will work in all windowses.
Please see documentation I posted in this ticket. #12356

[larsbergstrom]

See comments in #10665 (comment)