test_subprocess test_undecodable_env error on AIX #55402
Comments
|
The following tests fail in test_subprocess on AIX: ====================================================================== Traceback (most recent call last):
File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/test/test_subprocess.py", line 1039, in test_undecodable_env
self.assertEqual(stdout.decode('ascii'), ascii(value))
AssertionError: "'abc\\xff'" != "'abc\\udcff'"
- 'abc\xff'
? ^
+ 'abc\udcff'
? ^^^====================================================================== Traceback (most recent call last):
File "/san_cis/home/cis/.buildbot/python-aix6/3.x.phenix.xlc/build/Lib/test/test_subprocess.py", line 1039, in test_undecodable_env
self.assertEqual(stdout.decode('ascii'), ascii(value))
AssertionError: "'abc\\xff'" != "'abc\\udcff'"
- 'abc\xff'
? ^
+ 'abc\udcff'
? ^^^Ran 267 tests in 366.280s FAILED (failures=2, skipped=22) I haven't investigated yet. |
|
Perhaps related to the test_locale failure in bpo-11190. |
|
test_undecodable_code() in test_cmd_line had a similar issue: on FreeBSD, Solaris and Mac OS X, even if the locale is C (and nl_langinfo(CODESET) announces ASCII), _Py_char2wchar() (mbstowcs) decoded b'\xff' as '\xff' (use ISO-8859-1 encoding). To fix the test, I just patched the test to accept both results: b'\xff' may be decoded as '\udcff' or '\xff'. test_undecodable_env does something like: os.environb[b'test']=b'abc\xff', but os.getenv() decodes b'abc\xff' as 'abc\xff' instead of 'abc\udcff' even if LC_ALL=C. I don't understand why the test pass on FreeBSD, Solaris and Mac OS X, but not on AIX. It would be interesting to get the locale encoding of the child process using LC_ALL=C. |
Oh, the command line and the filesystem encodings may be different. test_undecodable_env() of test_subprocess.py uses os.environ which uses sys.getfilesystemencoding(), whereas test_undecodable_code() of test_cmd_line.py uses _Py_char2wchar() which uses mbstowcs() functions (which use the locale encoding). sys.getfilesystemencoding() is initialized from nl_langinfo(CODESET). nl_langinfo(CODESET) gives maybe "ISO-8859-1" (or an alias to this encoding) even if LC_ALL=C. |
|
I am not sure this is what you want: > LC_ALL=C ./python
Python 3.2rc3 (py3k:88417M, Feb 14 2011, 10:37:42)
[GCC 4.2.0] on aix6
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getfilesystemencoding()
'iso8859-1'
>>> What information can I provide to help solve this bug? |
>>> sys.getfilesystemencoding()
'iso8859-1'Ok, I expected this result. Can you also try: $ LC_ALL=C ./python -c "import sys; print(ascii(sys.argv))" $(echo -ne "abc\xff")
['-c', 'abc\udcff']
$ LC_ALL=C python3.1 -c "import locale; print(locale.nl_langinfo(locale.CODESET))"
ANSI_X3.4-1968
$ LC_ALL=C python3.1 -c "import locale; print(locale.getpreferredencoding())"
ANSI_X3.4-1968Anyway, test_undecodable_env() is not written to support ISO-8859-1 filesystem encoding. I should patch the test to check the filesystem encoding. |
$ LANG=C ./python -Wd -E -bb -c "import sys; print(ascii(sys.argv))" $(echo -ne "abc\xff")['-c', 'abc\xff']
['-c', 'abc\xff']
$ LANG=C ./python -Wd -E -bb -c "import locale; print(locale.nl_langinfo(locale.CODESET))"
ISO8859-1
$ LANG=C ./python -Wd -E -bb -c "import locale; print(locale.getpreferredencoding())"
ISO8859-1 |
|
ping myself |
DavidEdelsohn
mannequin
added
interpreter-core
|
Seems to be okay at least with Python 3.6 "test" version - but maybe there is better way to call these tests - just to be sure the one needed is not being skipped. On AIX 5.3 TL7 SP0: root@x064:[/data/prj/aixtools/python/python-3.6.0.162/Lib/test]../../python test_subprocess.py OK (skipped=24) On AIX 6.1 TL9 SP4: michael@x071:[/data/prj/aixtools/python/python-3.6.0.162/Lib/test]../../python test_subprocess.py OK (skipped=24) Please note that GNU libiconv is used by default (version 1.14) rather than IBM iconv. This may be the key issue (will test 11190 as well) |
|
Short Version: == Tests result: SUCCESS == 1 test OK. Total duration: 2 min 18 sec ... OK (skipped=33) == Tests result: SUCCESS == 1 test OK. Total duration: 2 min 16 sec Posting a PR - tested manually on AIX 5.3, AIX 6.1 and AIX 7.1 imho: being able to remove the special condition for AIX is an improvement. diff --git a/Lib/test/test_subprocess.py b/Lib/test/test_subprocess.py
index 73b57b21db..4719773b67 100644
--- a/Lib/test/test_subprocess.py
+++ b/Lib/test/test_subprocess.py
@@ -2228,15 +2228,9 @@ class POSIXProcessTestCase(BaseTestCase):
env = os.environ.copy()
env[key] = value
# Use C locale to get ASCII for the locale encoding to force
- # surrogate-escaping of \xFF in the child process; otherwise it can
- # be decoded as-is if the default locale is latin-1.
+ # surrogate-escaping of \xFF in the child process
env['LC_ALL'] = 'C'
- if sys.platform.startswith("aix"):
- # On AIX, the C locale uses the Latin1 encoding
- decoded_value = encoded_value.decode("latin1", "surrogateescape")
- else:
- # On other UNIXes, the C locale uses the ASCII encoding
- decoded_value = value
+ decoded_value = value
stdout = subprocess.check_output(
[sys.executable, "-c", script],
env=env) |
|
someone with modern AIX access reopen if this specific issue still appears to exist after the 3.8 and 3.7 PRs finish merging. |
gpshead
changed the title
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: