Empty PYTHONIOENCODING is not the same as nonexistent

[serhiy-storchaka]

BPO	18818
Nosy	@ncoghlan, @vstinner, @serhiy-storchaka
Files	empty_pythonioencoding.patch empty_pythonioencoding_2.patch empty_pythonioencoding_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-09-13.08:48:41.769>
created_at = <Date 2013-08-23.06:45:48.549>
labels = ['interpreter-core', 'type-feature']
title = 'Empty PYTHONIOENCODING is not the same as nonexistent'
updated_at = <Date 2014-02-25.15:55:52.592>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2014-02-25.15:55:52.592>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2013-09-13.08:48:41.769>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2013-08-23.06:45:48.549>
creator = 'serhiy.storchaka'
dependencies = []
files = ['31433', '31435', '31447']
hgrepos = []
issue_num = 18818
keywords = ['patch', 'needs review']
message_count = 9.0
messages = ['195946', '195947', '195950', '195951', '196007', '197501', '197560', '197561', '212186']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'vstinner', 'Arfrever', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18818'
versions = ['Python 3.4']

[serhiy-storchaka]

$ PYTHONIOENCODING= ./python
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: 
Aborted (core dumped)

As a consequence we can't set only the error handler.

$ PYTHONIOENCODING=:surrogateescape ./python
Fatal Python error: Py_Initialize: can't initialize sys standard streams
LookupError: unknown encoding: 
Aborted (core dumped)

Here is a patch which allows accepting empty encoding as default encoding.

[ncoghlan]

Patch looks good to me.

[vstinner]

Patch looks good to me, but you have to update Doc/using/cmdline.rst,
at least to add a versionchanged section.

Tests would also be nice :-)

I really like the the "PYTHONIOENCODING=:surrogateescape" use case!

[serhiy-storchaka]

Second variant of the patch also supports empty error handler as the default error handler (i.e. strict).

$ PYTHONIOENCODING=ascii: ./python
Python 3.4.0a1+ (default:5b5ef012cd4e+, Aug 23 2013, 10:18:51) 
[GCC 4.6.3] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print('\xbd')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xbd' in position 0: ordinal not in range(128)

Without it different error is raised:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: unknown error handler name ''

[serhiy-storchaka]

Here is a patch with tests and documentation. For the documentation I hope on your help.

[serhiy-storchaka]

Nick, could you help me with documentation? I'm sure my wording needs improving.

And there is one question left. Is this change appropriate for 2.7 and 3.3, or only for 3.4?

[python-dev]

New changeset c7fdb0637d0b by Serhiy Storchaka in branch 'default':
Issue bpo-18818: The "encodingname" part of PYTHONIOENCODING is now optional.
http://hg.python.org/cpython/rev/c7fdb0637d0b

[serhiy-storchaka]

Thank you Ezio and Vajrasky for the review.

[python-dev]

New changeset 8ac9c3754d33 by R David Murray in branch 'default':
whatsnew: encoding is now optional in PYTHONIOENCODING (bpo-18818)
http://hg.python.org/cpython/rev/8ac9c3754d33