regression: PySys_SetArgvEx(0, NULL, 0): SystemError: Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function

[hroncok]

BPO	37926
Nosy	@vstinner, @hroncok
PRs	bpo-37926: Fix PySys_SetArgvEx(0, NULL, 0) #15415 [3.8] bpo-37926: Fix PySys_SetArgvEx(0, NULL, 0) crash (GH-15415) #15420

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-23.11:23:48.102>
created_at = <Date 2019-08-23.07:39:09.430>
labels = ['extension-modules', '3.8', '3.9']
title = 'regression: PySys_SetArgvEx(0, NULL, 0): SystemError: Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function'
updated_at = <Date 2019-09-05.14:43:15.499>
user = 'https://github.com/hroncok'

bugs.python.org fields:

activity = <Date 2019-09-05.14:43:15.499>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-23.11:23:48.102>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2019-08-23.07:39:09.430>
creator = 'hroncok'
dependencies = []
files = []
hgrepos = []
issue_num = 37926
keywords = ['patch']
message_count = 6.0
messages = ['350262', '350268', '350269', '350275', '350276', '351201']
nosy_count = 2.0
nosy_names = ['vstinner', 'hroncok']
pr_nums = ['15415', '15420']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37926'
versions = ['Python 3.8', 'Python 3.9']

[hroncok]

There is a regression between Python 3.7 and 3.8 when using PySys_SetArgvEx(0, NULL, 0).

Consider this example:

#include <Python.h>

int main() {
    Py_Initialize();
    PySys_SetArgvEx(0, NULL, 0);  /* HERE */
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    Py_FinalizeEx();
    return 0;
}

This works in 3.7 but no longer does in 3.8:

$ gcc $(python3.7-config --cflags --ldflags) example.c 
$ ./a.out 
Today is Fri Aug 23 07:59:52 2019

$ gcc $(python3.8-config --cflags --ldflags --embed) example.c 
$ ./a.out 
Fatal Python error: no mem for sys.argv
SystemError: /builddir/build/BUILD/Python-3.8.0b3/Objects/unicodeobject.c:2089: bad argument to internal function

Current thread 0x00007f12c78b9740 (most recent call first):
Aborted (core dumped)

The documentation https://docs.python.org/3/c-api/init.html#c.PySys_SetArgvEx explicitly mentions passing 0 to PySys_SetArgvEx:

"if argc is 0..."

So I guess this is not something you shouldn't do.

[vstinner]

Oops, it's my fault! PR 15415 fix the crash.

The regression was introduced by:

commit 74f6568
Author: Victor Stinner <vstinner@redhat.com>
Date: Fri Mar 15 15:08:05 2019 +0100

bpo-36301: Add _PyWstrList structure (GH-12343)

Simplified change:

•    static wchar_t \*empty_argv[1] = {L""};
  

+ wchar_t* empty_argv[1] = {L""};

It's a deliberate change to not waste memory: PySys_SetArgvEx() (make_sys_argv()) is only called once, whereas static keeps the memory for the whole lifecycle of the process.

But I didn't notice a subtle issue with memory allocated on the stack: the code works well with gcc -O0! The bug only triggers with gcc -O3.

[vstinner]

New changeset c486825 by Victor Stinner in branch 'master':
bpo-37926: Fix PySys_SetArgvEx(0, NULL, 0) crash (GH-15415)
c486825

[vstinner]

New changeset ca9ae94 by Victor Stinner in branch '3.8':
bpo-37926: Fix PySys_SetArgvEx(0, NULL, 0) crash (GH-15415) (GH-15420)
ca9ae94

[vstinner]

I tested my fix manually using example from msg350262 and gcc -O3 (./configure && make): I can reproduce the crash without the change, and I confirm that my change fix the crash.

Thanks Miro for the bug report: the fix will be included in next Python 3.8 beta release. I fixed the bug in 3.8 and master branches (3.7 is not affected).

[vstinner]

FYI this bug was found in paraview in Fedora Rawhide: https://bugzilla.redhat.com/show_bug.cgi?id=1743896