Trivial mingw compile fixes

[jonny]

BPO	10615
Nosy	@loewis, @amauryfa, @merwok
Superseder	bpo-45538: MinGW is unsupported - close all open issues and list them here.
Files	Python-3.1.3.fix_mingw_compile.diff pyconfig.h

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-20.12:59:07.118>
created_at = <Date 2010-12-03.12:45:42.816>
labels = ['build', '3.8', '3.9', '3.10']
title = 'Trivial mingw compile fixes'
updated_at = <Date 2021-10-21.10:04:27.018>
user = 'https://bugs.python.org/jonny'

bugs.python.org fields:

activity = <Date 2021-10-21.10:04:27.018>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.12:59:07.118>
closer = 'iritkatriel'
components = ['Build']
creation = <Date 2010-12-03.12:45:42.816>
creator = 'jonny'
dependencies = []
files = ['19924', '19952']
hgrepos = []
issue_num = 10615
keywords = ['patch']
message_count = 22.0
messages = ['123236', '123302', '123318', '123320', '123386', '123391', '123396', '123397', '123404', '123405', '123407', '123409', '123463', '123464', '123475', '123489', '123523', '123526', '123530', '123578', '185821', '193253']
nosy_count = 6.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'schmir', 'eric.araujo', 'rpetrov', 'jonny']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '45538'
type = 'compile error'
url = 'https://bugs.python.org/issue10615'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[jonny]

Some C files fail to compile with mingw, mostly due to missing preprocessor statements:

• Modules/_ctypes/callproc.c
  When using the bundled non-msvc libffi, ffi_call has no return value, so only use it when compiled with msvc.
• Modules/_pickle.c
  There are Windows data types FLOAT, INT, LONG which the pickle code conflicts with.
• Modules/posixmodule.c
  Some "#ifdef _MSC_VER" paragraphs apply for __MINGW32__ as well; for MinGW, wincrypt.h must be explicitely included (the "original" Windows headers include them in windows.h)
• Modules/signalmodule.c
  On case-sensitive filesystems, windows.h must be all-lowercase
• Modules/socketmodule.h
  MinGW does not define SIO_KEEPALIVE_VALS, so do it here if required
• PC/msvcrtmodule.c
  MinGW does not have crtdbg.h
• PC/winreg.c
  MinGW does not define REG_LEGAL_CHANGE_FILTER, so do it here if required
• Python/dynload_win.c
  strcasecmp is already defined with MinGW, so use a private name
• Python/pythonrun.c
  MinGW has locale.h, but not langinfo.h

This patch fixes all the mentioned problems.

[loewis]

When the patch is applied, what's the resulting status of mingw compilation?

[jonny]

When the patch is applied, what's the resulting status of mingw compilation?

It compiles all C files which I require. Not sure if this is really *all* C files, but at least very close to all. I will post a list of object files I get on Monday.

[loewis]

Am 04.12.2010 09:32, schrieb Johann Hanne:

Johann Hanne <python@jf.hanne.name> added the comment:

> When the patch is applied, what's the resulting status of mingw compilation?

It compiles all C files which I require. Not sure if this is really *all* C files, but at least very close to all. I will post a list of object files I get on Monday.

Will it then also link something?

[jonny]

Will it then also link something?

Sure - it actually builds a python.exe which is fully working for me. If you need a proof, please let me know, I have no problem uploading it somewhere...

Is there a reason you are so sceptical? After all, Python for Linux is also usually compiled with gcc, so why shouldn't it work for Win32?

[loewis]

Sure - it actually builds a python.exe which is fully working for me. If you need a proof, please let me know, I have no problem uploading it somewhere...

Is there a reason you are so sceptical? After all, Python for Linux is also usually compiled with gcc, so why shouldn't it work for Win32?

There is a long track of people contributing patches to Python saying
"you need this and that to make it work on platform XYZ". Right after
accepting these patches (without testing, because we might not have
access to XYZ) they come back with more patches. If you then look
carefully at the initial submission, they never actually claimed that
this is *all* patches that are needed. Instead, they use phrases like
"some trivial fixes" (meaning that the difficult fixes are yet to come).

[jonny]

Ok, I see. And no, this is *not* my intention. I will post the list of successfully compiled objects files and the linker result on Monday as promised. If it turns out I lied, feel free to put me into the hall of shame.

After all, the Win32 platform is not that uncommon. I know, MinGW could be seen as a different platform, but I never encountered a situation where I required anything but the exe file (no special MinGW DLL or something like that).

[loewis]

After all, the Win32 platform is not that uncommon. I know, MinGW
could be seen as a different platform, but I never encountered a
situation where I required anything but the exe file (no special
MinGW DLL or something like that).

Traditionally, MingW-compiled Python binaries would often be
binary-incompatible with the ones available from python.org, since the
MingW build would use a different version of the MS CRT than the
python.org version. As a consequence, extensions built for the official
binaries would crash in the MingW build, and vice versa.

[amauryfa]

As a consequence, extensions built for the official
binaries would crash in the MingW build, and vice versa

IMO this is a use case for the new "soabi" tag...

[amauryfa]

Sure - it actually builds a python.exe which is fully working for me.
If you need a proof, please let me know, I have no problem uploading
it somewhere...

This is useless, of course. The build must be reproducible.
Here is what I did so far:

• Open a cmd.exe console, ensure that msys/bin and mingw/bin are in the PATH
• Run "sh configure" - it's slow, but works.
• Run "make"
• Re-running "make -k" fails on three files: import.c, posixmodule.c, and pwdmodule.c

Your patch does not touch import.c, and the pwd module should probably be removed from a Python running on Windows.
Do you use another method to compile Python?

[loewis]

IMO this is a use case for the new "soabi" tag...

Unfortunately not: these tags are not supported on Windows.

[amauryfa]

> IMO this is a use case for the new "soabi" tag...
Unfortunately not: these tags are not supported on Windows.
So it's a argument for adding (optional?) soabi tags on Windows!

[jonny]

Traditionally, MingW-compiled Python binaries would often be
binary-incompatible

I don't plan to distribute a MinGW compiled package as an alternative to the official MSVC based build. In my particular use case, I compile everything into a single exe and put the Lib/*.py files into a single zip file. The result is a two-file Python distribution which doesn't need an installation. If somebody is interested, I can elaborate, but this would be off-topic here.

As for the reproducibility issue: The configure/Makefile system coming with Python simply doesn't support creating a Windows build. I hope it's clear that the patch is NOT about creating a Windows installer. If this is a requirement for getting my patch in, please let me know so I can stop wasting my time!

Whatsoever, if you simply want to reproduce that my compile (!) fixes work, please follow these steps:

• Start a MinGW shell and cd into the Python source tree
• CFLAGS="-IPC -IPython" ./configure
• Remove the created pyconfig.h and replace PC/pyconfig.h with the manually modified version I'm attaching
• Modify the Makefile:
  -- Remove pwdmodule.c from the MODOBJS variable
  -- Add to the MODOBJS variable: PC/dl_nt.o PC/import_nt.o
  -- Remove the LIBOBJS variable assigment
  -- Edit the "# Build the interpreter" paragraph: Add PC/empty.o to the rule dependencies *AND* after "...$(SYSLIBS) $(LDLAST)"
• Modify PC/empty.c: Add "#include "Python.h" as first line; replace 'return 0;' by 'wchar_t *wargv[] = { L"python.exe", NULL }; return Py_Main(1, wargv);'
• Run make
• Add the the full path to the Lib directory to your PYTHONPATH

[jonny]

As promised, here comes the list of modules which successfully compile with MinGW:

./Python-3.1.3/PC/import_nt.o
./Python-3.1.3/PC/_subprocess.o
./Python-3.1.3/PC/dl_nt.o
./Python-3.1.3/PC/msvcrtmodule.o
./Python-3.1.3/PC/winreg.o
./Python-3.1.3/Parser/pgen.o
./Python-3.1.3/Parser/grammar1.o
./Python-3.1.3/Parser/parser.o
./Python-3.1.3/Parser/listnode.o
./Python-3.1.3/Parser/parsetok.o
./Python-3.1.3/Parser/tokenizer.o
./Python-3.1.3/Parser/printgrammar.o
./Python-3.1.3/Parser/myreadline.o
./Python-3.1.3/Parser/firstsets.o
./Python-3.1.3/Parser/metagrammar.o
./Python-3.1.3/Parser/grammar.o
./Python-3.1.3/Parser/acceler.o
./Python-3.1.3/Parser/node.o
./Python-3.1.3/Parser/bitset.o
./Python-3.1.3/Python/getcompiler.o
./Python-3.1.3/Python/codecs.o
./Python-3.1.3/Python/pystrtod.o
./Python-3.1.3/Python/pyfpe.o
./Python-3.1.3/Python/getargs.o
./Python-3.1.3/Python/modsupport.o
./Python-3.1.3/Python/getopt.o
./Python-3.1.3/Python/formatter_unicode.o
./Python-3.1.3/Python/mysnprintf.o
./Python-3.1.3/Python/pyctype.o
./Python-3.1.3/Python/sysmodule.o
./Python-3.1.3/Python/errors.o
./Python-3.1.3/Python/pymath.o
./Python-3.1.3/Python/pythonrun.o
./Python-3.1.3/Python/pystrcmp.o
./Python-3.1.3/Python/pystate.o
./Python-3.1.3/Python/getversion.o
./Python-3.1.3/Python/_warnings.o
./Python-3.1.3/Python/Python-ast.o
./Python-3.1.3/Python/frozen.o
./Python-3.1.3/Python/getcopyright.o
./Python-3.1.3/Python/asdl.o
./Python-3.1.3/Python/pyarena.o
./Python-3.1.3/Python/importdl.o
./Python-3.1.3/Python/compile.o
./Python-3.1.3/Python/dtoa.o
./Python-3.1.3/Python/getplatform.o
./Python-3.1.3/Python/traceback.o
./Python-3.1.3/Python/ceval.o
./Python-3.1.3/Python/mystrtoul.o
./Python-3.1.3/Python/thread.o
./Python-3.1.3/Python/symtable.o
./Python-3.1.3/Python/structmember.o
./Python-3.1.3/Python/future.o
./Python-3.1.3/Python/graminit.o
./Python-3.1.3/Python/peephole.o
./Python-3.1.3/Python/marshal.o
./Python-3.1.3/Python/import.o
./Python-3.1.3/Python/bltinmodule.o
./Python-3.1.3/Python/dynload_win.o
./Python-3.1.3/Python/ast.o
./Python-3.1.3/Modules/sha512module.o
./Python-3.1.3/Modules/socketmodule.o
./Python-3.1.3/Modules/expat/xmltok.o
./Python-3.1.3/Modules/expat/xmlrole.o
./Python-3.1.3/Modules/expat/xmlparse.o
./Python-3.1.3/Modules/_csv.o
./Python-3.1.3/Modules/_randommodule.o
./Python-3.1.3/Modules/errnomodule.o
./Python-3.1.3/Modules/_weakref.o
./Python-3.1.3/Modules/arraymodule.o
./Python-3.1.3/Modules/symtablemodule.o
./Python-3.1.3/Modules/zipimport.o
./Python-3.1.3/Modules/sha256module.o
./Python-3.1.3/Modules/mmapmodule.o
./Python-3.1.3/Modules/_ctypes/_ctypes_test.o
./Python-3.1.3/Modules/_ctypes/libffi/src/prep_cif.o
./Python-3.1.3/Modules/_ctypes/libffi/src/x86/ffi.o
./Python-3.1.3/Modules/_ctypes/libffi/src/x86/win32.o
./Python-3.1.3/Modules/_ctypes/callbacks.o
./Python-3.1.3/Modules/_ctypes/stgdict.o
./Python-3.1.3/Modules/_ctypes/_ctypes.o
./Python-3.1.3/Modules/_ctypes/callproc.o
./Python-3.1.3/Modules/_ctypes/malloc_closure.o
./Python-3.1.3/Modules/_ctypes/cfield.o
./Python-3.1.3/Modules/operator.o
./Python-3.1.3/Modules/posixmodule.o
./Python-3.1.3/Modules/md5module.o
./Python-3.1.3/Modules/timemodule.o
./Python-3.1.3/Modules/_functoolsmodule.o
./Python-3.1.3/Modules/datetimemodule.o
./Python-3.1.3/Modules/cmathmodule.o
./Python-3.1.3/Modules/zlibmodule.o
./Python-3.1.3/Modules/_threadmodule.o
./Python-3.1.3/Modules/itertoolsmodule.o
./Python-3.1.3/Modules/_lsprof.o
./Python-3.1.3/Modules/_tkinter.o
./Python-3.1.3/Modules/getbuildinfo.o
./Python-3.1.3/Modules/_collectionsmodule.o
./Python-3.1.3/Modules/_localemodule.o
./Python-3.1.3/Modules/_io/iobase.o
./Python-3.1.3/Modules/_io/bytesio.o
./Python-3.1.3/Modules/_io/stringio.o
./Python-3.1.3/Modules/_io/_iomodule.o
./Python-3.1.3/Modules/_io/textio.o
./Python-3.1.3/Modules/_io/bufferedio.o
./Python-3.1.3/Modules/_io/fileio.o
./Python-3.1.3/Modules/_struct.o
./Python-3.1.3/Modules/_codecsmodule.o
./Python-3.1.3/Modules/selectmodule.o
./Python-3.1.3/Modules/rotatingtree.o
./Python-3.1.3/Modules/sha1module.o
./Python-3.1.3/Modules/unicodedata.o
./Python-3.1.3/Modules/parsermodule.o
./Python-3.1.3/Modules/xxsubtype.o
./Python-3.1.3/Modules/_heapqmodule.o
./Python-3.1.3/Modules/audioop.o
./Python-3.1.3/Modules/_multiprocessing/win32_functions.o
./Python-3.1.3/Modules/_multiprocessing/semaphore.o
./Python-3.1.3/Modules/_multiprocessing/pipe_connection.o
./Python-3.1.3/Modules/_multiprocessing/multiprocessing.o
./Python-3.1.3/Modules/_multiprocessing/socket_connection.o
./Python-3.1.3/Modules/_bisectmodule.o
./Python-3.1.3/Modules/gcmodule.o
./Python-3.1.3/Modules/_sre.o
./Python-3.1.3/Modules/_json.o
./Python-3.1.3/Modules/mathmodule.o
./Python-3.1.3/Modules/signalmodule.o
./Python-3.1.3/Modules/pyexpat.o
./Python-3.1.3/Modules/binascii.o
./Python-3.1.3/Modules/_pickle.o
./Python-3.1.3/Objects/unicodeobject.o
./Python-3.1.3/Objects/enumobject.o
./Python-3.1.3/Objects/codeobject.o
./Python-3.1.3/Objects/methodobject.o
./Python-3.1.3/Objects/moduleobject.o
./Python-3.1.3/Objects/cobject.o
./Python-3.1.3/Objects/exceptions.o
./Python-3.1.3/Objects/floatobject.o
./Python-3.1.3/Objects/unicodectype.o
./Python-3.1.3/Objects/bytesobject.o
./Python-3.1.3/Objects/weakrefobject.o
./Python-3.1.3/Objects/fileobject.o
./Python-3.1.3/Objects/structseq.o
./Python-3.1.3/Objects/memoryobject.o
./Python-3.1.3/Objects/dictobject.o
./Python-3.1.3/Objects/classobject.o
./Python-3.1.3/Objects/setobject.o
./Python-3.1.3/Objects/object.o
./Python-3.1.3/Objects/obmalloc.o
./Python-3.1.3/Objects/cellobject.o
./Python-3.1.3/Objects/descrobject.o
./Python-3.1.3/Objects/tupleobject.o
./Python-3.1.3/Objects/typeobject.o
./Python-3.1.3/Objects/genobject.o
./Python-3.1.3/Objects/iterobject.o
./Python-3.1.3/Objects/listobject.o
./Python-3.1.3/Objects/bytearrayobject.o
./Python-3.1.3/Objects/frameobject.o
./Python-3.1.3/Objects/boolobject.o
./Python-3.1.3/Objects/longobject.o
./Python-3.1.3/Objects/funcobject.o
./Python-3.1.3/Objects/complexobject.o
./Python-3.1.3/Objects/rangeobject.o
./Python-3.1.3/Objects/capsule.o
./Python-3.1.3/Objects/bytes_methods.o
./Python-3.1.3/Objects/abstract.o
./Python-3.1.3/Objects/sliceobject.o

[amauryfa]

Thanks for the explanations! These instructions should be turned into a patch.
Some questions already:

• Why is the generated pyconfig.h not good enough? Could we modify the ./configure script instead?
• the "empty.c" should remain empty... Why not use PC/WinMain.c instead?
• why is a WinMain required when python.c is supposed to define a main()?
  Is the final binary a console application or a Windows application?

[loewis]

As for the reproducibility issue: The configure/Makefile system
coming with Python simply doesn't support creating a Windows build. I
hope it's clear that the patch is NOT about creating a Windows
installer. If this is a requirement for getting my patch in, please
let me know so I can stop wasting my time!

That's why I was asking what the patch achieves - it was not clear
at all what the objective is. Not supporting the build of an installer
is fine (although that probably wouldn't be difficult). However, if
the patch doesn't manage to make "configure;make" work, it's IMO
useless. Having "make install" work in some form would be desirable.

[jonny]

• Why is the generated pyconfig.h not good enough? Could we modify the ./configure script instead?
  Probably setup.py needs to be extended; I did not have a deep look into the build system though

• the "empty.c" should remain empty... Why not use PC/WinMain.c instead?
  It seems to be that MinGW does not support the wWinMain unicode entry point; the empty.c hack is just to show that it's actually linking

• why is a WinMain required when python.c is supposed to define a
  main()?
  Good question; probably just a question of linker flags; I will have a look if my time permits

• Is the final binary a console application or a Windows application?
  A Console application; creating a GUI application is easily done with the "-mwindows" gcc flag, but then you will have no Python console when not using idle or similar

[jonny]

However, if the patch doesn't manage to make "configure;make" work, it's IMO useless.
No, it's not, for two reasons:

• Embedding Python by just compiling/linking all the .c files in seems to be a major feature to me; so fixing compilation is useful for its own
• The win32 build system has never used "configure;make", but a Visual Studio project file; so why require it for a MinGW build?!

Having "make install" work in some form would be desirable.
It would be a "nice to have". But there is one thing: The patch in its current form is trivial and next to impossible to break anything, yet I'm sure it's useful for a number of people. It's a result of work done within the company I'm working for. Submitting the patch does not have an immediate benefit for my company, it only will save me a (short!) amount of time because I will not to have to re-apply the patch for every new Python release. This time saving is the only justification for spending some working time to try to get it into the official tree.

As I have the feeling that all the discussion (which for the major part I used my spare time already...) is taking too much time, I will stop doing it as part of my job. As I'm privately interested in open source projects, I will continue to have a look at fixing "configure;make" for MinGW; but given my very limited spare time, this might take some weeks...

[jonny]

Two corrections for the "configure;make" workaround:

• WRONG: Remove the created pyconfig.h and replace PC/pyconfig.h with the manually modified version I'm attaching
• CORRECT: Remove PC/pyconfig.h and replace the created pyconfig.h with the manually modified version I'm attaching
• Modules/config.c needs to modified as well:
  -- Replace PyInit_posix by PyInit_nt (two occurences), "posix" by "nt" (in _PyImport_Inittab)
  -- Remove '{"pwd", PyInit_pwd},' in _PyImport_Inittab

[loewis]

• Embedding Python by just compiling/linking all the .c files in
  seems to be a major feature to me; so fixing compilation is useful
  for its own

If that's the objective of the patch, I'm -1 on it.

• The win32 build system has never used "configure;make", but a
  Visual Studio project file; so why require it for a MinGW build?!

Well, if MingW could use the VS project files, that would be fine
with me as well... We need *some* build procedure. Just being
able to compile the source files is not maintainable.

It would be a "nice to have". But there is one thing: The patch in
its current form is trivial and next to impossible to break anything,
yet I'm sure it's useful for a number of people. It's a result of
work done within the company I'm working for. Submitting the patch
does not have an immediate benefit for my company, it only will save
me a (short!) amount of time because I will not to have to re-apply
the patch for every new Python release. This time saving is the only
justification for spending some working time to try to get it into
the official tree.

Understood. Perhaps somebody else is interested in picking up the
patch.

BTW, you do have your employer's permission to contribute this work,
right?

[BreamoreBoy]

bpo-17605 titled "mingw-meta: build interpeter core" was opened on 2013-03-31 so I believe this can be closed as superseded.

[rpetrov]

This patch cannot be closed as contain some fixes not reported/proposed yet .
For instance I'm not aware of name clash in Modules/_pickle.c .
I think that other are in scope of bpo-3871 or earlier .