unicodeobject can coredump on exit

[rsalz]

BPO	528620
Nosy	@mwhudson, @malemburg, @loewis

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/malemburg'
closed_at = <Date 2002-04-08.15:35:08.000>
created_at = <Date 2002-03-11.19:39:00.000>
labels = ['interpreter-core']
title = 'unicodeobject can coredump on exit'
updated_at = <Date 2002-04-08.15:35:08.000>
user = 'https://bugs.python.org/rsalz'

bugs.python.org fields:

activity = <Date 2002-04-08.15:35:08.000>
actor = 'lemburg'
assignee = 'lemburg'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2002-03-11.19:39:00.000>
creator = 'rsalz'
dependencies = []
files = []
hgrepos = []
issue_num = 528620
keywords = []
message_count = 9.0
messages = ['9635', '9636', '9637', '9638', '9639', '9640', '9641', '9642', '9643']
nosy_count = 4.0
nosy_names = ['mwh', 'lemburg', 'loewis', 'rsalz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue528620'
versions = ['Python 2.2']

[rsalz]

It looks like the internal unicode_empty object can be
destroyed before other unicode objects are destroyed.

    /* Convert to Unicode */
    if (len == 0) {
***     if (unicode_empty == NULL) goto onError;
        Py_INCREF(unicode_empty);
        v = (PyObject *)unicode_empty;
    }
    else
        v = PyUnicode_Decode(s, len, encoding, errors);

The line marked *** was added. It prevents python from
segfaulting during its cleanup-and-exit phase.

[malemburg]

Logged In: YES
user_id=38388

Provided you only use Unicode objects with the
Unicode implementation properly initialized, this
cannot happen. unicode_empty is kept alive between
calls to _PyUnicode_Init() and _PyUnicode_Fini().

If you are seeing a core dump in the location
you suggested, then it's likely that you have hit
a ref count bug somewhere.

In any case, I'd need an example to be able to say
whether this is indeed a bug in the core or in some
extension module.

[rsalz]

Logged In: YES
user_id=36737

Running python 2.1.1 against ZSI, for example test/t3.py segfaults on exit with the following as top of stack.
(gdb) bt
#0 0x0809365c in PyUnicode_FromEncodedObject () at eval.c:41
#1 0x0809352d in PyUnicode_FromObject () at eval.c:41
#2 0x0809714f in PyUnicode_Compare () at eval.c:41
#3 0x0808cb5d in PyObject_Unicode () at eval.c:41
#4 0x0808cf75 in PyObject_Compare () at eval.c:41
#5 0x0808d129 in PyObject_RichCompare () at eval.c:41
#6 0x08057361 in PyEval_EvalCode () at eval.c:41
#7 0x0805914f in PyEval_CallObjectWithKeywords () at eval.c:41
#8 0x08057829 in PyEval_EvalCode () at eval.c:41
#9 0x0805914f in PyEval_CallObjectWithKeywords () at eval.c:41
#10 0x08057829 in PyEval_EvalCode () at eval.c:41
#11 0x0805914f in PyEval_CallObjectWithKeywords () at eval.c:41
#12 0x08057829 in PyEval_EvalCode () at eval.c:41
#13 0x0805914f in PyEval_CallObjectWithKeywords () at eval.c:41
#14 0x08057829 in PyEval_EvalCode () at eval.c:41
#15 0x0805914f in PyEval_CallObjectWithKeywords () at eval.c:41
#16 0x08057829 in PyEval_EvalCode () at eval.c:41
#17 0x0805905b in PyEval_CallObjectWithKeywords () at eval.c:41
#18 0x08058c90 in PyEval_CallObjectWithKeywords () at eval.c:41
#19 0x08058f2b in PyEval_CallObjectWithKeywords () at eval.c:41
#20 0x08058c7d in PyEval_CallObjectWithKeywords () at eval.c:41
#21 0x08058b73 in PyEval_CallObjectWithKeywords () at eval.c:41
#22 0x0807c749 in PyInstance_New () at eval.c:41
---Type <return> to continue, or q <return> to quit---

#23 0x0808a0e4 in PyDict_New () at eval.c:41
#24 0x0808a3b6 in PyDict_SetItem () at eval.c:41
#25 0x0808bfc4 in _PyModule_Clear () at eval.c:41
#26 0x080654d6 in PyImport_Cleanup () at eval.c:41
#27 0x0806ae72 in Py_Finalize () at eval.c:41
#28 0x08051fae in Py_Main () at eval.c:41
#29 0x40080507 in __libc_start_main (main=0x8051a20 <main>, argc=2,
    ubp_av=0xbffff9c4, init=0x8050e7c <_init>, fini=0x809f070 <_fini>,
    rtld_fini=0x4000dc14 <_dl_fini>, stack_end=0xbffff9bc)
    at ../sysdeps/generic/libc-start.c:129
(gdb)

[loewis]

Logged In: YES
user_id=21627

Notice that unicode_empty is accessed from within
PyImport_Cleanup, which is currently run after PyUnicode_Fini.

I believe PyUnicode_Fini should be run *much* later, e.g.
after PyFloat_Fini.

[rsalz]

Logged In: YES
user_id=36737

The original report described a test case; test/t3.py in the
ZSI package. Following up on this, Dave Wallace has a short
example that crashes. See his posting in to xml-sig, at
http://mail.python.org/pipermail/xml-sig/2002-April/007553.html

[malemburg]

Logged In: YES
user_id=38388

Moving PyUnicode_Fini down in the finalization list
seems to help.

Still, I'm afraid that if some Unicode objects are left
stored in some lists or dictionaries, we may still see
the problem (due to coercion of strings to Unicode).
OTOH, strings are finalized before Unicode object now.

Anyway, the test case works now after checkin
of the fix proposed by Martin, so I'm closing the
report.

[mwhudson]

Logged In: YES
user_id=6656

So, do we want this in 221? I know you may have said in a
checkin message...

[loewis]

Logged In: YES
user_id=21627

It's not a critical bug that is being fixed (a crash is
critical, but it is a rare problem).

OTOH, the fix seems reasonably straight-forward, and low
risk (he says :-); so yes - if that is still possible, it
should go into 2.2.1. Otherwise, it is a clear 2.2.2 candidate.

[malemburg]

Logged In: YES
user_id=38388

I marked it as "2.2.1 candidate" group. IMHO, this should
be done with all minor update candidates (I only found this
group thingie today -- seems like a nice way to markup
bug fix candidates).