Compilation on MINGW64 fails (CODESET,wcstok,...)

[vengelson]

BPO	37801
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba, @lazka, @eamanu, @erikjanss
Superseder	bpo-45538: MinGW is unsupported - close all open issues and list them here.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-20.12:59:58.042>
created_at = <Date 2019-08-09.11:53:21.190>
labels = ['interpreter-core', '3.8', '3.9', '3.10', 'build', 'OS-windows']
title = 'Compilation on MINGW64 fails (CODESET,wcstok,...)'
updated_at = <Date 2021-10-21.10:04:33.640>
user = 'https://bugs.python.org/vengelson'

bugs.python.org fields:

activity = <Date 2021-10-21.10:04:33.640>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-20.12:59:58.042>
closer = 'iritkatriel'
components = ['Build', 'Interpreter Core', 'Windows']
creation = <Date 2019-08-09.11:53:21.190>
creator = 'vengelson'
dependencies = []
files = []
hgrepos = []
issue_num = 37801
keywords = []
message_count = 6.0
messages = ['349282', '349283', '349294', '349312', '349337', '349900']
nosy_count = 9.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'lazka', 'eamanu', 'erikjanss', 'vengelson']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = '45538'
type = 'compile error'
url = 'https://bugs.python.org/issue37801'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[vengelson]

Compilation on MINGW64 fails (CODESET,wcstok,...)
I am using the latest MINGW64 (http://repo.msys2.org/distrib/x86_64/msys2-x86_64-20190524.exe)
Versions: Python-3.7.2, Python-3.8.0b3

$ gcc -v
Using built-in specs.
COLLECT_GCC=C:\msys64\mingw64\bin\gcc.exe
COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.1.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
gcc version 9.1.0 (Rev3, Built by MSYS2 project)

Result of make:

Python/initconfig.c: In function 'config_get_locale_encoding':
Python/initconfig.c:1427:28: error: implicit declaration of function 'nl_langinfo' [-Werror=implicit-function-declaration]
1427 | const char *encoding = nl_langinfo(CODESET);
| ^~~~~~~~~~~
Python/initconfig.c:1427:40: error: 'CODESET' undeclared (first use in this function); did you mean 'ECONNRESET'?
1427 | const char *encoding = nl_langinfo(CODESET);
| ^~~~~~~
| ECONNRESET
Python/initconfig.c:1427:40: note: each undeclared identifier is reported only once for each function it appears in
Python/initconfig.c: In function 'config_init_env_warnoptions':
Python/initconfig.c:1992:18: error: too many arguments to function 'wcstok'
1992 | # define WCSTOK wcstok
| ^~~~~~
Python/initconfig.c:2015:20: note: in expansion of macro 'WCSTOK'
2015 | for (warning = WCSTOK(env, L",", &context);
| ^~~~~~
In file included from ./Include/Python.h:30,
from Python/initconfig.c:1:
C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h:147:20: note: declared here
147 | wchar_t *cdecl wcstok(wchar_t * __restrict _Str,const wchar_t * __restrict__ _Delim) __MINGW_ATTRIB_DEPRECATED_SEC_WARN;
| ^~~~~~
Python/initconfig.c:1992:18: error: too many arguments to function 'wcstok'
1992 | # define WCSTOK wcstok
| ^~~~~~
Python/initconfig.c:2017:20: note: in expansion of macro 'WCSTOK'
2017 | warning = WCSTOK(NULL, L",", &context))
| ^~~~~~
In file included from ./Include/Python.h:30,
from Python/initconfig.c:1:
C:/msys64/mingw64/x86_64-w64-mingw32/include/string.h:147:20: note: declared here
147 | wchar_t *cdecl wcstok(wchar_t * __restrict _Str,const wchar_t * __restrict__ _Delim) __MINGW_ATTRIB_DEPRECATED_SEC_WARN;
| ^~~~~~
cc1.exe: some warnings being treated as errors
make: *** [Makefile:1703: Python/initconfig.o] Error 1

[vengelson]

(code from Python-3.8.0b3, but Python-3.7.2 had similar issues)

[zooba]

Guessing it needs an extra header file that is implicitly included in the Windows headers?

[eryksun]

I wasn't aware that CPython builds for MSYS2 out of the box, since it's a POSIX-on-Windows platform like Cygwin. Apparently there are patches that enable it to build, since MSYS2 has Python available.

For Windows, WCSTOK expands to wcstok_s, which takes a context pointer that allows concurrently parsing multiple strings in a single thread. The old function lacks this parameter and instead uses a per-thread static buffer. They're declared as follows:

wchar_t *wcstok(wchar_t *strToken, const wchar_t *strDelimit);
wchar_t *wcstok_s(wchar_t *str, const wchar_t *delimiters,
                  wchar_t **context);

Otherwise the WCSTOK macro expands to wcstok, which assumes that POSIX systems use the standard definition [1]:

wchar_t *wcstok(wchar_t *restrict ws1, const wchar_t *restrict ws2,
                wchar_t **restrict ptr);

Apparently the version of wcstok declared in your build environment takes only two arguments, like the old insecure function in Windows:

wchar_t *__cdecl wcstok(wchar_t * __restrict__ _Str,
                        const wchar_t * __restrict__ _Delim)
__MINGW_ATTRIB_DEPRECATED_SEC_WARN;

[1] http://pubs.opengroup.org/onlinepubs/9699919799/functions/wcstok.html

[lazka]

I wasn't aware that CPython builds for MSYS2 out of the box, since it's a POSIX-on-Windows platform like Cygwin. Apparently there are patches that enable it to build, since MSYS2 has Python available.

MSYS2 consists of a cygwin like environment and a mingw one. OP is trying to build with mingw.

MSYS2 contains a somewhat heavily patched Python in both environments so it's expected that building plain CPython doesn't work. There are plans to upstream smaller non-MSYS2 specific patches but nothing has come of it so far.

[erikjanss]

fyi 1 : this issue pops up in multiple places, cfr :

• bpo-35890
• bpo-20596

the selection of the wcstok function is based on MS_WINDOWS being
defined, rather than eg. an autoconf check on which function is
available.

fyi 2 : I've been able to cross compile 3.8 with mingw64 (gcc 7.3), but with a custom meson script instead of autoconf