Mingw32/64 support

[gabomdq]

I'm using Greenlet in my project which cross compiles from Linux to Windows using Mingw. For Mingw in 32 bit mode it works almost out of the box, but I had to comment out the Visual Studio entries in slp_platformselect.h, so ideally a check for some MINGW flags in the first two entries would be needed, or perhaps a ! defined(_MSC_VER)

Anyway, in 32 bit mode, after commenting out the MSVC entries everything works, switch_x86_unix.h is used, and task switching is flawless AFAICT.

However, in 64 bit mode, when switch_amd64_unix.h is selected, a compilation error is produced:

/tmp/ccFWucFb.s: Assembler messages:
/tmp/ccFWucFb.s:1937: Error: operand type mismatch for add' /tmp/ccFWucFb.s:1938: Error: operand type mismatch foradd'

I've also noticed that switch_x86_unix.h has special ifdef'ed WIN32 sections that are not present in switch_amd64_unix.h, I'm not sure if these are required as well.

[snaury]

@gabomdq I'll check if there's anything broken with latest MinGW versions, but bear in mind that:

1. 32-bit linux and windows have almost identical ABIs, so switch_x86_unix.h can be used
2. greenlet was actually ported to MinGW

Unfortunately 64-bit Windows has ABI that is completely different from anything else, hence you'd need a separate port. We've done one for Microsoft Visual C++, but there's no MinGW port at the moment. In 64-bit Windows case you cannot just add features to switch_amd64_unix.h.

Another point is that it's not entirely clear how to setup 64-bit MinGW properly: it doesn't seem to be officially supported and there are several TDM versions, all of which were hard to setup and only half-working last time I tried (admittedly 1 or 2 years ago, so maybe something changed, I haven't rechecked since then).

[gabomdq]

Thanks for the reply. For what it's worth, I'm using Ubuntu's official packages for Mingw, which install through the meta package mingw-w64.

[snaury]

@gabomdq you could try using switch_x64_masm.obj (linking with it should probably work with MinGW, you may have to rename it as switch_x64_mask.o though), then force it to use switch_x64_msvc.h (there should be nothing msvc specific in there) and make sure both MS_WIN64 and _M_X64 are defined, which would probably work.

Also, could you share info on how to replicate your setup? Then I could make a VM here and try to work on it, but I need to know how do you cross-compile Python extensions (do you cross-compile Python, and how?), since I've never done that before.

[gabomdq]

Cool, I'll try it out.

As to my system, I'm using Greenlet in my Python/Cython engine, you can check out the code here: https://bitbucket.org/gabomdq/ignifuga

With a Ubuntu 12.04 64 bits VM, after checking out the code from Bitbucket, you can run (in tools)

./schafer -D

which will install the build dependencies, and then

./schafer -P win32

That will build a Python interpreter inside dist/bin for Win32, which will hold Greenlet among many other things. If you try to run

./schafer -P win64 you'll get an error I just added, but you can comment those lines (starting on line 863 in schafer.py) and it'll proceed until at the very end you get the errors I posted in my first message.

If you look at the external/greenlet sources in there you'll see I patched up the slp_platformselect.h header file to skip the MSVC parts.

[ghost]

I don't know if my problem belongs to this issue but I am not able to install greenlet via mingw64. Windows shouldn't be the problem because I was able to install it via the Windows console.

I am getting the following messages:

$ pip3 install greenlet
Collecting greenlet
  Downloading http://debieks01.fft-it.de/nexus/repository/pypi/packages/67/62/ca2a95648666eaa2ffeb6a9b3964f21d419ae27f82f2e66b53da5b943fc4/greenlet-0.4.10.zip (82kB)
Installing collected packages: greenlet
  Running setup.py install for greenlet: started
    Running setup.py install for greenlet: finished with status 'error'
    Complete output from command C:/msys64/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;__file__='C:/Users/bo32782/AppData/Local/Temp/pip-build-k0xxhy48/greenlet/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:/Users/bo32782/AppData/Local/Temp/pip-myk25fr1-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'greenlet' extension
    creating build
    creating build/temp.mingw-3.5
    C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe -Wno-unused-result -Wsign-compare -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG -IC:/msys64/mingw64/include/python3.5m -c greenlet.c -o build/temp.mingw-3.5/greenlet.o
    {standard input}: Assembler messages:
    {standard input}:518: Error: operand type mismatch for `add'
    {standard input}:519: Error: operand type mismatch for `add'
    error: command 'C:\\msys64\\mingw64\\bin/x86_64-w64-mingw32-gcc.exe' failed with exit status 1

    ----------------------------------------
Command "C:/msys64/mingw64/bin/python3.exe -u -c "import setuptools, tokenize;__file__='C:/Users/bo32782/AppData/Local/Temp/pip-build-k0xxhy48/greenlet/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record C:/Users/bo32782/AppData/Local/Temp/pip-myk25fr1-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in C:/Users/bo32782/AppData/Local/Temp/pip-build-k0xxhy48/greenlet/

Seems to be a 64 bit problem to me... Any ideas ?

[snaury]

Sorry, I haven't been using Windows for many years now, and don't know what's currently new with mingw, nobody did a 64-bit mingw port as far as I now. However I'm not sure why you would use it with Python in this day and age anyway? There's Visual C++ for Python: https://www.microsoft.com/en-us/download/details.aspx?id=44266 if you're on 2.7, and latest Visual Studios have Express editions capable of compiling Python extensions. There are binary wheels on PyPI too if you don't want to setup a compiler.

[iongion]

@snaury it is true it is a hard to see use case, probably will get solved once and for all with with https://msdn.microsoft.com/en-us/commandline/wsl/faq. Now to explain, I for one use Windows as primary workstation and there are some initiatives such as msys2 using pacman that really look promising in bringing the power of homebrew from the mac world to windows people. But there, the transition is much smoother due to their Unix nature. We all understand this is not for production time, this is only for development, that will do the job. I am trying to develop a python-socket-io app and having a lot of UI tools on screen for debugging is really helpful. Windows helps in this area as it is so generous in what exists on the market, you can find a tool for almost any task.

[snaury]

@iongion I'm not sure what you're saying, there are plenty of use cases for Windows and greenlet already works on Windows, both x86 and x64. It's just that there's no x64 port for mingw or cygwin/msys2, so people need to either use a native toolchain, or need to work on a port if it's pressing enough.

If there's a good quality pull request with the port I would gladly accept it. This issue is open for 4 years now and nobody stepped up yet.

[iongion]

@snaury I would gladly do-it, it just came back for me :) but I have no idea where to start due to the fact that it requires advanced understanding of compilers, platforms and CPU architectures.

Investigating, it fails when running this:

gcc -v -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG -DNDEBUG -IW:/SDKs/msys2/mingw64/include/python2.7 -c greenlet.c -o build/temp.mingw-2.7/greenlet.o

with

 W:/SDKs/msys2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/6.2.0/../../../../x86_64-w64-mingw32/include
End of search list.
GNU C11 (Rev2, Built by MSYS2 project) version 6.2.0 (x86_64-w64-mingw32)
        compiled by GNU C version 6.2.0, GMP version 6.1.1, MPFR version 3.1.4-p3, MPC version 1.0.3, isl version 0.15
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 94423b12c802d0182374f038ff49eeb8
{standard input}: Assembler messages:
{standard input}:483: Error: operand type mismatch for `add'
{standard input}:484: Error: operand type mismatch for `add'

It seems an ASM issue for this case - but I am unable to identify from what source file

• Python 2.7.12 64 bit
• gcc 6.2.0

gcc -v
Using built-in specs.
COLLECT_GCC=W:\SDKs\msys2\mingw64\bin\gcc.exe
COLLECT_LTO_WRAPPER=W:/SDKs/msys2/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/6.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-6.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,objc,obj-c++,fortran,ada --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev2, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld
Thread model: posix
gcc version 6.2.0 (Rev2, Built by MSYS2 project)

Any tips on where to start looking ?

[ghost]

I met this issue while installing bpython via pip3 install bpython under msys2/mingw64. I built it, but no way sure I am that it really works (but tests passed OK).

For readers who try to install bpython: it's probably not worth it, since bpython requires fcntl and termios anyway. These are missing on windows by design, because of unability to port from unix.

I substituted fcntl via hack and could dig further, but lost all courage on the way. I had to go ahead or to write this comment. Sorry.

---

For other people who are curious about how to compile greenlet-0.4.12 under msys2/mingw64, here it is:

All following command go into your mingw64 shell:

$ cd ~
$ pip3 download greenlet
$ tar xzf greenlet-0.4.12.tar.gz

Apply this patch for greenlet-0.4.12/greenlet.c:

--- greenlet-0.4.12/greenlet.c  2017-02-01 22:43:08 +0300
+++ greenlet-0.4.12-n/greenlet.c        2017-05-24 05:01:49 +0300
@@ -1,5 +1,6 @@
 /* vim:set noet ts=8 sw=8 : */

+#define EXTERNAL_ASM
 #define GREENLET_MODULE

 #include "greenlet.h"

Apply this patch for greenlet-0.4.12/platform/switch_amd64_unix.h:

--- greenlet-0.4.12/platform/switch_amd64_unix.h        2016-12-11 02:52:06 +0300
+++ greenlet-0.4.12-n/platform/switch_amd64_unix.h      2017-05-24 05:02:03 +0300
@@ -44,7 +44,8 @@
     void* rbx;
     unsigned int csr;
     unsigned short cw;
-    register long *stackref, stsizediff;
+    register long *stackref;
+    register long long stsizediff;
     __asm__ volatile ("" : : : REGS_TO_SAVE);
     __asm__ volatile ("fstcw %0" : "=m" (cw));
     __asm__ volatile ("stmxcsr %0" : "=m" (csr));

Patches are trivial; if you don't know how to patch, edit sources by hand.

Let's test it (optional):

$ cd ~/greenlet-0.4.12
$ python3 setup.py test
...
Ran 63 tests in 0.791s

OK

Pack it back and install by hand:

$ cd ~
$ rm greenlet-0.4.12.tar.gz
$ tar czf greenlet-0.4.12.tar.gz greenlet-0.4.12
$ pip3 install greenlet-0.4.12.tar.gz              # we pass filename, not package name

---

To greenlet authors: the issue, if it resolved right, seems to be much simpler than...

In 64-bit Windows case you cannot just add features to switch_amd64_unix.h.

Another point is that it's not entirely clear how to setup 64-bit MinGW properly: it doesn't seem to be officially supported and there are several TDM versions, all of which were hard to setup and only half-working last time I tried (admittedly 1 or 2 years ago, so maybe something changed, I haven't rechecked since then).

Maybe I simply passed tests through scariest ABI UB ever, I can't tell. Changes were obvious:

• It generated addq %eax, %rsp, probably because sizeof(long) is 32-bit. Implementation seems to set it stsizediff to ts_target->stack_start - (char*)stackref, and that is actually ptrdiff_t, not long, but maybe this was intended to build on ansi, idk. If not, ptrdiff_t is the right choice. grep -rn '\<long\>' . gives only platfrom/switch* files, so main source is probably safe in that regard.
• It could not resolve slp_save_state_asm etc at link stage, so I enabled it right in the main source (it must be passed as -DEXTERNAL_ASM to the compiler, but I'm not familiar with setup.py, sorry).

Anyway, MSYS2 is definitely the current way to use MinGW (and real half of unix) on windows. Original MinGW & MSYS are dead for too long. Not sure about TDM.

I cannot reinstall MSYS2 only for this issue, but I think these MSYS shell commands reproduce it:

$ pacman -Syu                       # per msys2 installation manual
...close via [X] and restart this shell...
$ pacman -Su                        # per msys2 installation manual
# now msys2 is installed into c:/msys64, updated properly and is ready to work

$ pacman -S mingw-w64-x86_64-toolchain
$ pacman -S mingw-w64-x86_64-python3
$ pacman -S mingw-w64-x86_64-python3-pip

And then in mingw64 shell:

$ pip3 install greenlet

If you encapsulate these changes in production-ready form into a branch and link to it, I'll test it in few hours or a day after, no need for complex setups.

[snaury]

You seem to be both defining EXTERNAL_ASM (which is only needed if platform/switch_x64_masm.obj is being used) and changing switch_amd64_unix.h, which doesn't make a lot of sense. What's probably happening is that switch_amd64_unix.h is saving just enough registers to pass tests, but more subtle breakage may happen in real world usage, e.g. rdi and rsi are not among saved registers there, plus windows x64 abi has callee saved xmm registers, so floating point might lead to problems.

Does it work for you if you replace #include "slp_platformselect.h" with #include "platform/switch_x64_msvc.h"?

[ghost]

Sorry for the late answer. It seems that build is likely misconfigured for mingw, because switch_x64_masm.obj is linked, but the assembler error appears in switch_amd64_unix.h. I didn't touch anything in build process. Maybe switch_amd64_unix.h is not used after compilation at all? But it is used too: adding ud2 to that asm chunk makes first test fail with "Illegal instruction" error.

Does it work for you if you replace #include "slp_platformselect.h" with #include "platform/switch_x64_msvc.h"?

One warning, tests ok.

$ python3 setup.py build
running build
running build_ext
building 'greenlet' extension
C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe -Wno-unused-result -Wsign-compare -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fwrapv -D__USE_MINGW_ANSI_STDIO=1 -DNDEBUG -IC:/msys64/mingw64/include/python3.5m -c greenlet.c -o build/temp.mingw-3.5/greenlet.o
In file included from greenlet.c:328:0:
platform/switch_x64_msvc.h:25:0: warning: "alloca" redefined
 #define alloca _alloca

In file included from C:/msys64/mingw64/x86_64-w64-mingw32/include/stdlib.h:695:0,
                 from C:/msys64/mingw64/include/python3.5m/Python.h:34,
                 from greenlet.h:8,
                 from greenlet.c:5:
C:/msys64/mingw64/x86_64-w64-mingw32/include/malloc.h:183:0: note: this is the location of the previous definition
 #define alloca(x) __builtin_alloca((x))

writing build/temp.mingw-3.5/greenlet-cpython-35m.def
creating build/lib.mingw-3.5
C:\msys64\mingw64\bin/x86_64-w64-mingw32-gcc.exe -shared -Wl,--enable-auto-image-base -pipe -s -s build/temp.mingw-3.5/greenlet.o platform/switch_x64_masm.obj build/temp.mingw-3.5/greenlet-cpython-35m.def -LC:/msys64/mingw64/lib/python3.5/config-3.5m -L/mingw64/lib -lpython3.5m -lversion -o build/lib.mingw-3.5/greenlet-cpython-35m.dll

$ python3 setup.py test
running test
running egg_info
creating greenlet.egg-info
writing top-level names to greenlet.egg-info/top_level.txt
writing greenlet.egg-info/PKG-INFO
writing dependency_links to greenlet.egg-info/dependency_links.txt
writing manifest file 'greenlet.egg-info/SOURCES.txt'
reading manifest file 'greenlet.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'greenlet.egg-info/SOURCES.txt'
running build_ext
Linking C:/msys64/home/User/greenlet-0.4.12/build/lib.mingw-3.5/greenlet-cpython-35m.dll to C:/msys64/home/User/greenlet-0.4.12/greenlet-cpython-35m.dll
...
...
...
----------------------------------------------------------------------
Ran 63 tests in 0.735s

OK

[iongion]

On latest msys usr/include/python3.6m/pyport.h:219:10: fatal error: sys/select.h: No such file or directory

pip3 install greenlet
Collecting greenlet
  Using cached greenlet-0.4.12.tar.gz
Installing collected packages: greenlet
  Running setup.py install for greenlet ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-7itnv3p0/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-zzzo6q_t-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    building 'greenlet' extension
    creating build
    creating build/temp.mingw64_nt-10.0-2.9.0(0.318
    creating build/temp.mingw64_nt-10.0-2.9.0(0.318/5
    creating build/temp.mingw64_nt-10.0-2.9.0(0.318/5/3)-x86_64-3.6
    x86_64-pc-msys-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=x86-64 -mtune=generic -O2 -pipe -I/usr/include/python3.6m -c greenlet.c -o build/temp.mingw64_nt-10.0-2.9.0(0.318/5/3)-x86_64-3.6/greenlet.o
    In file included from W:/SDKs/msys2/usr/include/python3.6m/Python.h:50:0,
                     from greenlet.h:8,
                     from greenlet.c:5:
    W:/SDKs/msys2/usr/include/python3.6m/pyport.h:219:10: fatal error: sys/select.h: No such file or directory
     #include <sys/select.h>
              ^~~~~~~~~~~~~~
    compilation terminated.
    error: command 'x86_64-pc-msys-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-7itnv3p0/greenlet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-zzzo6q_t-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-7itnv3p0/greenlet/

[DancingQuanta]

I have been trying to install neovim in msys2 which depends on greenlet. I got a similar error as the previous post. Perhaps sys/select.h is not available for windows as suggested here?

[snaury]

Well, greenlet does not include sys/select.h, I would suggest reporting it to the project that is actually responsible for that include and which could maybe help you (I would assume that is either cpython directly or whoever is responsible for msys2 port of cpython).