sys.argv docs should explaining how to handle encoding issues

[ncoghlan]

BPO	17110
Nosy	@ncoghlan, @pitrou, @ezio-melotti, @methane, @manueljacob, @miss-islington
PRs	bpo-17110: doc: add note how to get bytes from sys.argv #12602 [3.7] bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602) #12626
Files	Issue17110.patch: Documentation for proper encoding of command line arguments.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-30.06:25:04.085>
created_at = <Date 2013-02-03.04:01:11.615>
labels = ['3.8', 'type-feature', '3.7', 'expert-unicode', 'docs']
title = 'sys.argv docs should explaining how to handle encoding issues'
updated_at = <Date 2020-06-18.11:19:04.439>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2020-06-18.11:19:04.439>
actor = 'methane'
assignee = 'docs@python'
closed = True
closed_date = <Date 2019-03-30.06:25:04.085>
closer = 'methane'
components = ['Documentation', 'Unicode']
creation = <Date 2013-02-03.04:01:11.615>
creator = 'ncoghlan'
dependencies = []
files = ['34470']
hgrepos = []
issue_num = 17110
keywords = ['patch']
message_count = 11.0
messages = ['181239', '213674', '213699', '213911', '214022', '339175', '339176', '371778', '371788', '371802', '371806']
nosy_count = 10.0
nosy_names = ['ncoghlan', 'pitrou', 'ezio.melotti', 'Arfrever', 'methane', 'docs@python', 'mjacob', 'sreepriya', 'andyma', 'miss-islington']
pr_nums = ['12602', '12626']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17110'
versions = ['Python 3.7', 'Python 3.8']

[ncoghlan]

The sys.argv docs [1] currently contain no mention of the fact that they are Unicode strings decoded from bytes provided by the OS. They also don't explain how to correct a decoding error by reversing Python's implicit conversion and redoing it based on the application's knowledge of the correct encoding, as described at [2]

[1] http://docs.python.org/3/library/sys#sys.argv
[2] http://stackoverflow.com/questions/6981594/sys-argv-as-bytes-in-python-3k/

[sreepriya]

I tried running with Python 3.4 the following code

import sys

print(sys.argv[1])
print(b'bytes')

And I ran as follows trying to run with a different encoding.
$ python ~/a.py echo priya|iconv -t latin1
priya
bytes

There was no unicode encode error generated! Is it because the problem is fixed?

[pitrou]

There was no unicode encode error generated! Is it because the problem
is fixed?

No, it's not fixed.
First, it seems you are testing with Python 2 (otherwise you would get "b'bytes'", not "bytes"). Python 2 won't have a problem here, since it treats everything as bytestrings.
Second, to evidence the issue you must pass a non-ASCII string. For example:

$ ./python a.py `echo éléphant|iconv -t latin1`
Traceback (most recent call last):
  File "a.py", line 4, in <module>
    print(sys.argv[1])
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 0: surrogates not allowed

[sreepriya]

You are right. Instead of running ./python inside the python directory, I ran the default python of older version! Based on the stackoverflow link given, I tried to make some documentation. I am attaching the patch!

[pitrou]

Hmm, I'm not sure where those explanations belong but I'm not sure should be in the sys module docs (especially as they are quite lengthy, and they also apply to other data such as os.environ). Perhaps the Unicode HOWTO?

[methane]

New changeset 38f4e46 by Inada Naoki in branch 'master':
bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602)
38f4e46

[miss-islington]

New changeset 5b80cb5 by Miss Islington (bot) in branch '3.7':
bpo-17110: doc: add note how to get bytes from sys.argv (GH-12602)
5b80cb5

[manueljacob]

The actual startup code uses Py_DecodeLocale() for converting argv from bytes to unicode. Since which Python version is it guaranteed that Py_DecodeLocale() and os.fsencode() roundtrip?