Inconsistency between venv and site

[serhiy-storchaka]

BPO	25185
Nosy	@vsajip, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-10-01.10:28:53.013>
created_at = <Date 2015-09-19.21:29:19.923>
labels = ['type-bug', 'library']
title = 'Inconsistency between venv and site'
updated_at = <Date 2015-10-01.10:28:53.011>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-10-01.10:28:53.011>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-10-01.10:28:53.013>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2015-09-19.21:29:19.923>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 25185
keywords = []
message_count = 7.0
messages = ['251119', '251141', '251292', '251776', '251781', '251783', '252016']
nosy_count = 4.0
nosy_names = ['vinay.sajip', 'Arfrever', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25185'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

[serhiy-storchaka]

venv writes pyvenv.cfg with utf-8 encoding, but site reads it with locale encoding.

[serhiy-storchaka]

Example:

$ LC_ALL=ru_RU.cp1251 ./python -m venv ../vpy×
$ LC_ALL=ru_RU.cp1251 ./python -m venv ../vpyØ
Error: 'charmap' codec can't encode character '\udc98' in position 617: character maps to <undefined>

The first command is successful, the latter command is failed. 'Ø'.encode('utf-8') == b'\xc3\x98', and 0x98 is invalid code in CP1251.

[vsajip]

Seems like the right fix would be to change site.py to read it using UTF-8.

[serhiy-storchaka]

I suspect that this can break backward compatibility.

[vsajip]

I suspect that this can break backward compatibility.

But since venv writes it out as UTF-8 already, it should be read in UTF-8, and if it isn't, isn't that the cause of the error which led to this bug we're talking about?

If we now write it with locale encoding, anything that previously assumed we were writing it in UTF-8 might break. While I'm not aware of any specific software which reads pyvenv.cfg other than the site.py code, there could well be third party code which does so.

[serhiy-storchaka]

I thought that pyvenv.cfg was used by venv's predecessor virtualenv. Now I see that I was wrong and your idea is right.

[python-dev]

New changeset d927c6cae05f by Vinay Sajip in branch '3.4':
Closes bpo-25185: Use UTF-8 encoding when reading pyvenv.cfg.
https://hg.python.org/cpython/rev/d927c6cae05f

New changeset eaf9220bdee3 by Vinay Sajip in branch '3.5':
Closes bpo-25185: merged fix from 3.4.
https://hg.python.org/cpython/rev/eaf9220bdee3

New changeset 69dd42cef190 by Vinay Sajip in branch 'default':
Closes bpo-25185: merged fix from 3.5.
https://hg.python.org/cpython/rev/69dd42cef190