test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal

[voidspace]

BPO	8589
Nosy	@brettcannon, @vstinner, @bitdancer, @voidspace, @florentx
Files	sys_warnoptions_encoding.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2010-09-17.23:41:02.436>
created_at = <Date 2010-05-01.11:05:18.503>
labels = ['type-bug', 'tests']
title = 'test_warnings.CEnvironmentVariableTests.test_nonascii fails under an ascii terminal'
updated_at = <Date 2010-09-17.23:41:02.434>
user = 'https://github.com/voidspace'

bugs.python.org fields:

activity = <Date 2010-09-17.23:41:02.434>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2010-09-17.23:41:02.436>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2010-05-01.11:05:18.503>
creator = 'michael.foord'
dependencies = []
files = ['17389']
hgrepos = []
issue_num = 8589
keywords = ['patch', 'buildbot']
message_count = 19.0
messages = ['104709', '104716', '104718', '104721', '104774', '105895', '106010', '106012', '106013', '106054', '106079', '106093', '114695', '114697', '114703', '116057', '116106', '116153', '116742']
nosy_count = 5.0
nosy_names = ['brett.cannon', 'vstinner', 'r.david.murray', 'michael.foord', 'flox']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue8589'
versions = ['Python 3.2']

[voidspace]

If you run test_warnings.py under an ascii terminal (at least on Mac OS X) then test_warnings.CEnvironmentVariableTests.test_nonascii fails. Perhaps the test should be skipped?

======================================================================
FAIL: test_nonascii (main.CEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "Lib/test/test_warnings.py", line 758, in <module>
    test_main()
  File "Lib/test/test_warnings.py", line 753, in test_main
    PyEnvironmentVariableTests
  File "/compile/python-trunk3/Lib/test/support.py", line 1053, in run_unittest
    _run_suite(suite)
  File "/compile/python-trunk3/Lib/test/support.py", line 1027, in _run_suite
    result = runner.run(suite)
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 158, in run
    result.printErrors()
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 109, in printErrors
    self.printErrorList('FAIL', self.failures)
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 116, in printErrorList
    self.stream.writeln("%s" % err)
  File "/compile/python-trunk3/Lib/unittest/runner.py", line 24, in writeln
    self.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 121: ordinal not in range(128)

Looks like the traceback here actually comes from unittest trying to report the failure.

[bitdancer]

Victor has proposed a patch for the traceback problem for regrtest, I think. I haven't looked at it, but I wonder if there is something that can instead be done to make unittest work in cases like this when run in an ascii terminal. See bpo-8522.

[voidspace]

What does bpo-8522 have to do with it - did you mean a different issue? In unittest it could catch the UnicodeEncodeError and write the ascii repr instead.

[bitdancer]

Heh, yes, I meant bpo-8533.

[vstinner]

The bug only occurs on Mac OS X because file system encoding is hardcoded to UTF-8 on this OS and the test is skipped if the file system encoding is ASCII.

But the bug is not specific to test_warnings, as mentionned by R. David Murray: bpo-8533 is the real bug. test_warnings is just an example to reproduce it.

I just commited a fix for bpo-8533: it should fix also this issue.

[vstinner]

Does my fix this bpo-8533 fixes this issue or not? I guess yes, but I would prefer a confirmation.

[vstinner]

The test was introduced by r80066 (issue bpo-7301).

[vstinner]

os.environ is decoded with sys.getfilesystemencoding(), whereas PYTHONWARNINGS is decoded with the locale encoding (locale.getpreferredencoding()) :-/

[vstinner]

Attached patch creates PySys_AddWarnOptionUnicode() function and decodes PYTHONWARNINGS variable with the file system encoding (as it is done for os.environ, to be consistent) instead of the locale encoding.

The patch only concerns the POSIX version. Windows is not affected by this issue (see below).

--

This issue occurs if the file system encoding is different than the locale encoding (especially if the locale encoding is ASCII). It only occurs on Mac OS because on this OS, the file system encoding is hardcoded to UTF-8 whereas the locale encoding... depends on the locale.

I reproduced the bug on Linux by hardcoding the file system encoding to UTF-8.

--

About the test output (UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' ...): I think that Michael Foord executed directly Lib/test/test_warnings.py instead of using "Lib/test/regrtest.py -v test_warnings". I only patched regrtest.py to use backslashreplace error handler on sys.stdout.

[vstinner]

tarek tested on Mac OS X: the patch fixes test_warnings issue.

[vstinner]

Ok, I commited my patch: r81358 (py3k). I'm waiting for the buildbots before backporting the fix to 3.1.

[vstinner]

Oh, Python 3.1 doesn't use the PYTHONWARNINGS variable: commit blocked in 3.1 (r81362).

[florentx]

A similar issue occurs on "x86 debian parallel" buildbot:

======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home2/buildbot2/slave/3.x.loewis-parallel/build/Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:DeprecaciónWarning']".encode('utf-8'))
AssertionError: b'' != b"['ignore:Deprecaci\xc3\xb3nWarning']"

http://www.python.org/dev/buildbot/builders/x86%20debian%20parallel%203.x/builds/80

[florentx]

There's a failure on the same buildbot, "x86 debian parallel", with test test_subprocess (test_undecodable_env).

This test was introduced with issue bpo-8391.

The test_subprocess and test_warnings failures are specific to this buildbot.

[vstinner]

I know test_warnings failure and I will try to fix it soon.

[vstinner]

It should be fixed by r84694.

[florentx]

Still happens with r84709 on PPC Tiger 3.x

======================================================================
FAIL: test_nonascii (test.test_warnings.CEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\xc3\x83\xc2\xb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

======================================================================
FAIL: test_nonascii (test.test_warnings.PyEnvironmentVariableTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.parc-tiger-1/build/Lib/test/test_warnings.py", line 731, in test_nonascii
    "['ignore:Deprecaci\xf3nWarning']".encode('utf-8'))
AssertionError: b"['ignore:Deprecaci\xc3\x83\xc2\xb3nWarning']" != b"['ignore:Deprecaci\xc3\xb3nWarning']"

---

Ran 71 tests in 9.113s

FAILED (failures=2)

http://www.python.org/dev/buildbot/builders/PPC%20Tiger%203.x/builds/566

[vstinner]

Still happens with r84709 on PPC Tiger 3.x

It's not the same error, PYTHONWARNINGS is decoded from the wrong encoding: locale encodind instead of utf-8. r84731 should fix this bug (at least, it restores the encoding used because my last commit, r84694).

With r84731, Python decodes PYTHONWARNINGS with strict error handler on Mac OS X, instead of surrogateescape error handler. I don't think that it matters because PYTHONWARNINGS is not supposed to contain raw bytes, and anyway, Mac OS X rejects invalid utf-8 filenames.

[vstinner]

I don't see any test_warnings anymore on http://code.google.com/p/bbreport/wiki/PythonBuildbotReport. Close this issue.