Audit does not work with non-ASCII data on non-UTF-8 locale

[serhiy-storchaka]

BPO	41094
Nosy	@vstinner, @serhiy-storchaka, @zooba, @miss-islington
PRs	bpo-41094: Fix decoding errors with audit when open files. #21095 [3.9] bpo-41094: Fix decoding errors with audit when open files. (GH-21095) #21107 [3.8] bpo-41094: Fix decoding errors with audit when open files. (GH-21095) #21108 bpo-41094: Additional fix for PYTHONSTARTUP. #21119 [3.9] bpo-41094: Fix decoding errors with audit when open files. (GH-21095) #21120 [3.8] bpo-41094: Fix decoding errors with audit when open files. (GH-21095) #21121 [3.9] bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119) #21124 [3.8] bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119) #21125

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-06-24.17:06:51.203>
created_at = <Date 2020-06-23.20:25:16.306>
labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10']
title = 'Audit does not work with non-ASCII data on non-UTF-8 locale'
updated_at = <Date 2020-06-24.17:16:06.330>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2020-06-24.17:16:06.330>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-24.17:06:51.203>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2020-06-23.20:25:16.306>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 41094
keywords = ['patch']
message_count = 10.0
messages = ['372205', '372217', '372232', '372234', '372235', '372259', '372260', '372261', '372267', '372268']
nosy_count = 4.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'steve.dower', 'miss-islington']
pr_nums = ['21095', '21107', '21108', '21119', '21120', '21121', '21124', '21125']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41094'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[serhiy-storchaka]

There are issues with using PySys_Audit() with non-ASCII data on non-UTF-8 locale.

One example is with PYTHONSTARTUP. In pymain_run_startup() in Modules/main.c the value of the PYTHONSTARTUP environment variable is passed to PySys_Audit() as UTF-8 encoded data. If it contains non-ASCII characters and the locale encoding is different from UTF-8, it fails.

There are similar bugs in _Py_fopen() and _Py_fopen_obj().

[zooba]

SGTM. Thanks for the PR!

[serhiy-storchaka]

New changeset 6c6810d by Serhiy Storchaka in branch 'master':
bpo-41094: Fix decoding errors with audit when open files. (GH-21095)
6c6810d

[serhiy-storchaka]

Oh, I made a mistake in tests (they do not test with non-ASCII data), but fixed tests fail on Windows.

[serhiy-storchaka]

The problem is that they pass successfully on my local Windows, but fail on CI.

[miss-islington]

New changeset 7329c8c by Miss Islington (bot) in branch '3.9':
bpo-41094: Fix decoding errors with audit when open files. (GH-21095)
7329c8c

[miss-islington]

New changeset 1813d31 by Miss Islington (bot) in branch '3.8':
bpo-41094: Fix decoding errors with audit when open files. (GH-21095)
1813d31

[serhiy-storchaka]

New changeset a7dc714 by Serhiy Storchaka in branch 'master':
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119)
a7dc714

[miss-islington]

New changeset 1dda40c by Miss Islington (bot) in branch '3.8':
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119)
1dda40c

[miss-islington]

New changeset 29aa808 by Miss Islington (bot) in branch '3.9':
bpo-41094: Additional fix for PYTHONSTARTUP. (GH-21119)
29aa808