python doesn't build if prefix contains non-ascii characters

[zegreek]

BPO	6011
Nosy	@vstinner, @tarekziade, @merwok, @nphilipp, @dstufft
Dependencies	bpo-9841: sysconfig and distutils.sysconfig differ in subtle ways
Files	tb.txt: configure line and traceback text_file.diff: patch to distutils.text_file.py textio.diff: patch to Modules/_io/textio.c (alternative to the first) distutils_makefile_encoding.patch copy_script.patch copy_script-2.patch test_distutils_surrogateescape.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2014-03-24.21:44:06.329>
created_at = <Date 2009-05-13.11:01:48.448>
labels = ['type-bug', 'library']
title = "python doesn't build if prefix contains non-ascii characters"
updated_at = <Date 2014-03-24.21:44:06.327>
user = 'https://bugs.python.org/zegreek'

bugs.python.org fields:

activity = <Date 2014-03-24.21:44:06.327>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2014-03-24.21:44:06.329>
closer = 'vstinner'
components = ['Distutils']
creation = <Date 2009-05-13.11:01:48.448>
creator = 'zegreek'
dependencies = ['9841']
files = ['13974', '13975', '13984', '18840', '19551', '20165', '21992']
hgrepos = []
issue_num = 6011
keywords = ['patch']
message_count = 47.0
messages = ['87674', '87675', '87766', '87772', '114568', '114571', '116060', '116087', '116263', '116297', '116390', '116391', '119074', '119102', '119353', '119368', '119383', '119389', '119409', '119420', '119449', '119847', '119848', '119891', '119894', '120178', '120818', '120821', '120827', '120829', '121551', '121564', '121566', '121810', '124571', '124597', '124646', '124647', '124648', '124650', '124651', '127016', '135750', '135755', '135903', '214727', '214738']
nosy_count = 6.0
nosy_names = ['vstinner', 'tarek', 'eric.araujo', 'zegreek', 'nils', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6011'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[zegreek]

I have tried to build python (version 3.1 beta 1) on linux and install
it to a non-standard prefix which contains non-ascii utf-8 characters
(my locale being utf-8). The build directory's path is ascii-only. The
exact configure line is given in the attached file 'tb.txt'.

Then the 'make' command fails at the stage where python extensions are
built, with the traceback displayed in file tb.txt (in short:
UnicodeDecodeError: 'ascii' codec can't decode byte ... ).

The problem is triggered when 'distutils.sysconfig.get_config_vars'
tries to parse the Makefile. The Makefile is opened with
'distutils.text_file.TextFile', which in turns calls 'io.open' with no
'encoding' parameter. At this stage of the build, the 'locale' module is
not available (due to '_collections' not being), so that
'locale.getprefferedencoding' cannot be called and the encoding falls
back to ascii (a quick look to 'Modules/_io/textio.c' suggests that this
fallback mechanism is already designed for being used at build time).

The solution I propose would be to use 'sys.getfilesystemencoding' as a
fallback first, as it is defined during build time on most systems:
windows, mac and on posix if 'CODESET' exists in 'langinfo.h'. Given
that in build routines, non-ascii characters are only likely to be
encountered in filesystem paths, this seems a reasonable behavior.

The attached patch 'text_file.diff' implements this strategy in
'distutils.text_file', and then calls 'io.open' with the appropriate
'encoding' parameter. It could be argued, however, that this new
fallback is of general interest and should be implemented directly in
'Modules/_io/textio.c'. If you deem so, I could try to come up with a
new patch.

The attached patch solves the problem on my system, and does not
introduce test failures (which is expected, as the new fallback should
only make a difference at build time).

Cheers,
Baptiste

[zegreek]

And here comes the patch

[zegreek]

OK, here is also the patch to 'Modules/_io/textio.c', as it is in fact
quite trivial. Choose which one you prefer :-)

Baptiste

[tarekziade]

Thanks, I'll work on this during this week end hopefully

[vstinner]

This issue may be related to bpo-9561.

[vstinner]

Change _io.TextIOWrapper() heuristic to choose the encoding is a bad idea.

[vstinner]

New patch:

• add encoding option to TextFile constructor
• parse_makefile() uses the heuristic from text_file.diff

Note: sys.getfilesystemencoding() is always set in Python 3.2 (but it may be None in Python 2.x and Python < 3.2).

[zegreek]

Hello,

I just tried your patch on latest svn (r84707), but I found out that the problem I reported can no more be reproduced. First, '_locale' seems now to be built earlier. Also, a fallback has been introduced in 'locale.getpreferredencoding'. When '_locale' cannot be imported, the encoding is now parsed from the environment variables (cf 'Lib/locale.py', line 558 and below). It looks like 'locale.getpreferredencoding' is now no more likely to fail than 'sys.getfilesystemencoding'. So I'm not sure if a patch is still needed at all.

In case a patch still makes sense, pay attention that there now also is a 'Lib/sysconfig.py', which also has a '_parse_makefile' function. This function uses a logic similar to the one in 'Lib/distutils/sysconfig.py', even thought it uses the builtin 'open', so it would need the same fix, if one is needed.

Cheers, B.

[merwok]

Victor: I would like bpo-9841 to be solved first so that we’re sure what file(s) need to be patched (and where to add a test for parsing).

Baptiste: Can you try to reproduce your bug with 2.7 and 3.1?

Tarek: I can take this over if you want.

[zegreek]

Eric: the bug does not exist with 2.7, as the Makefile is read as bytes.
It exists with 3.1.2.

By the way, when I say the bug is solved on 3.2, I only mean the narrow
problem of using a
non-ascii prefix that *is* decodable with the current locale. I do not
mean the more general
problem that arises, for example, when building with the 'C' locale, as
is discussed in bpo-9561.
With a 'C' locale, the build fails also with 3.2.

[vstinner]

For non-ascii directory name but ascii locale (eg. C locale), we have 3 choices:
a- read Makefile as a binary file
b- use the PEP-383
c- refuse to compile

(a) doesn't seem easy because it looks like distutils use the unicode type for all paths. (b) supposes to patch distutils to ensure that reading (and writing?) Makefile uses errors='surrogateescape'.

About (c), it can be a temporary solution. But I also think that non-ascii directory name and ascii locale encoding is a rare use case.

[vstinner]

Warning: "use the PEP-383" may impact other distutils component because the path may be written into to other files, which mean that we have to use errors='surrogateescape' for these files too.