Python 3.6 cannot reopen .pyc file with non-ASCII path

[tianjg]

BPO	32381
Nosy	@pfmoore, @vstinner, @tjguk, @ezio-melotti, @zware, @eryksun, @zooba, @izbyshev, @ZackerySpytz
PRs	bpo-32381: .pyc files with non-ASCII paths cannot be reopened on Windows #14699 bpo-32381: Fix PyRun_SimpleFileExFlags() encoding #23642 [3.9] bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642) #23692 [3.8] bpo-32381: Fix PyRun_SimpleFileExFlags() encoding (GH-23642) (GH-23692) #23696 bpo-32381: Rewrite PyErr_ProgramText() #23700 bpo-32381: Add _PyRun_SimpleFileObject() #23709 bpo-32381: Remove unused _Py_fopen() function #23711 bpo-32381: Add _PyRun_AnyFileObject() #23723 bpo-32381: pymain_run_command() uses PyCF_IGNORE_COOKIE #23724 bpo-32381: pymain_run_file() uses PySys_FormatStderr() #23778
Files	20171218111240.jpg

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-12-08.17:12:35.886>
created_at = <Date 2017-12-20.06:07:24.842>
labels = ['interpreter-core', 'type-bug', '3.8', 'OS-windows', '3.10', 'expert-unicode', '3.9']
title = 'Python 3.6 cannot reopen .pyc file with non-ASCII path'
updated_at = <Date 2021-01-06.09:13:10.218>
user = 'https://bugs.python.org/tianjg'

bugs.python.org fields:

activity = <Date 2021-01-06.09:13:10.218>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-12-08.17:12:35.886>
closer = 'izbyshev'
components = ['Interpreter Core', 'Unicode', 'Windows']
creation = <Date 2017-12-20.06:07:24.842>
creator = 'tianjg'
dependencies = []
files = ['47341']
hgrepos = []
issue_num = 32381
keywords = ['patch', '3.6regression']
message_count = 27.0
messages = ['308705', '308707', '308709', '308723', '308831', '308832', '347896', '382502', '382503', '382504', '382509', '382510', '382512', '382514', '382734', '382743', '382750', '382751', '382752', '382770', '382776', '382804', '382806', '383063', '383067', '383653', '384477']
nosy_count = 11.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'eryksun', 'steve.dower', 'izbyshev', 'ZackerySpytz', 'Tianjg', 'tianjg']
pr_nums = ['14699', '23642', '23692', '23696', '23700', '23709', '23711', '23723', '23724', '23778']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32381'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[tianjg]

have a problem that python3.6 can not reopen .pyc file with Chinese path, and python3.5 can reopen the same pyc file. As shown in the picture

[eryksun]

run_file encodes the file path via PyUnicode_EncodeFSDefault, which encodes as UTF-8 in Windows, starting with 3.6. PyRun_SimpleFileExFlags subsequently tries to open this encoded path via _Py_fopen, which calls fopen. The CRT expects an ANSI encoded path, so only the common ASCII subset will work. Non-ASCII paths will fail.

This could be addressed in _Py_fopen by decoding the path and calling _wfopen instead of fopen.

Executing a .pyc also fails in 3.5 if the wide-character path can't be encoded as ANSI, but the 3.5 branch only accepts security fixes.

[tianjg]

Thanks a lot. What should I do to reopen .pyc file with non-ASCII path use
python3.6 in cmd？Could you give me* some **code examples*.Thank you again,
and I look forward to hearing from you

2017-12-20 15:35 GMT+08:00 Eryk Sun <report@bugs.python.org>:

Eryk Sun <eryksun@gmail.com> added the comment:

run_file encodes the file path via PyUnicode_EncodeFSDefault, which
encodes as UTF-8 in Windows, starting with 3.6. PyRun_SimpleFileExFlags
subsequently tries to open this encoded path via _Py_fopen, which calls
fopen. The CRT expects an ANSI encoded path, so only the common ASCII
subset will work. Non-ASCII paths will fail.

This could be addressed in _Py_fopen by decoding the path and calling
_wfopen instead of fopen.

Executing a .pyc also fails in 3.5 if the wide-character path can't be
encoded as ANSI, but the 3.5 branch only accepts security fixes.

----------
components: +Interpreter Core, Unicode
nosy: +eryksun, ezio.melotti, vstinner
stage: -> test needed
title: python3.6 can not reopen .pyc file with Chinese path -> Python 3.6
cannot reopen .pyc file with non-ASCII path
type: compile error -> behavior
versions: +Python 3.7

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue32381\>

---

[eryksun]

Workarounds: (1) force 3.6 to use the legacy ANSI filesystem encoding by setting the environment variable PYTHONLEGACYWINDOWSFSENCODING. (2) Use 8.3 DOS names, if creating them is enabled on the volume. You can check their value in CMD via dir /x. (3) Create alternative directory symbolic links or junctions with ASCII names via CMD's mklink command.

[vstinner]

run_file() gets a wchar_t* string which comes from wmain() argv.

run_file() encodes the wchar_t* using PyUnicode_EncodeFSDefault().

Later, PyRun_SimpleFileExFlags() calls indirectly fopen() with the encoded filename.

This could be addressed in _Py_fopen by decoding the path and calling _wfopen instead of fopen.

I agree that it's the correct fix.

I would make _Py_fopen() more compatible with the PEP-529.

[vstinner]

I would make _Py_fopen() more compatible with the PEP-529.

Typo: It* would