The os module should unset() environment variable at exit

[vstinner]

BPO	39395
Nosy	@vstinner, @ericsnowcurrently, @serhiy-storchaka
PRs	bpo-39395: Clear env vars set by Python at exit #18078 bpo-39395: os.putenv() is now always available #18135

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-24.13:12:45.258>
created_at = <Date 2020-01-20.11:58:54.480>
labels = ['interpreter-core', '3.9', 'type-crash']
title = 'The os module should unset() environment variable at exit'
updated_at = <Date 2020-01-24.20:35:09.542>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-01-24.20:35:09.542>
actor = 'eric.snow'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-24.13:12:45.258>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-01-20.11:58:54.480>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 39395
keywords = ['patch']
message_count = 10.0
messages = ['360309', '360368', '360369', '360471', '360472', '360515', '360530', '360614', '360615', '360654']
nosy_count = 3.0
nosy_names = ['vstinner', 'eric.snow', 'serhiy.storchaka']
pr_nums = ['18078', '18135']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39395'
versions = ['Python 3.9']

[vstinner]

os.environ[key] = value has to keep internally the string "key=value\0" after putenv("key=value\0") has been called, since the glibc doesn't copy the string. Python has to manage the string memory.

Internally, the posix module uses a "posix_putenv_garbage" dictionary mapping key (bytes) to value (bytes). Values are "key=value\0" strings.

The bpo-35381 issue converted the os ("posix" in practice) module PEP-384: "Remove all static state from posixmodule": commit b396663. The _posix_clear() function is now called by _PyImport_Cleanup().

Problem: the glibc is not aware that Python is exiting and that the memory of the environment variable has been released. Next access to environment variables ("environ" C variable, putenv, setenv, unsetenv, ...) can crash. Sometimes, it doesn't crash even if the memory has been released, because free() does not always dig immediately holes in the heap memory (the complex problelm of memory fragmentation).

The posix module should notify the glibc that the memory will be released before releasing the memory, to avoid keeping dangling pointers in the "environ" C variable.

The following crash in the Elements module is an example of crash introduced by commit b396663 which introduced this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1791761

[vstinner]

Fedora downstream issue, crash in the "elements" package: https://bugzilla.redhat.com/show_bug.cgi?id=1791761

[vstinner]

I proposed bpo-39406 which avoids to have to clear environment variables set by Python at exit on platforms providing setenv().

[serhiy-storchaka]

bpo-39406 is a new feature. In any case we should fix potential crash in older Python versions.

[vstinner]

bpo-39406 is a new feature. In any case we should fix potential crash in older Python versions.

Python 3.8 and older are not affected by this issue since they never destroy the posix_putenv_garbage dictionary: memory is never released.