_elementtree.TreeBuilder broken with a non-C-deriving element_factory

[tiran]

BPO	16089
Nosy	@birkenfeld, @pitrou, @larryhastings, @tiran, @ezio-melotti, @skrah, @florentx, @mitya57
Files	issue16089-pseudo-fix.diff ctree.patch ctree2.patch c_treebuilder.patch c_treebuilder2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-10-04.17:59:30.043>
created_at = <Date 2012-09-30.02:07:42.686>
labels = ['extension-modules', 'expert-XML', 'type-crash', 'release-blocker']
title = '_elementtree.TreeBuilder broken with a non-C-deriving element_factory'
updated_at = <Date 2012-11-09.13:51:34.072>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2012-11-09.13:51:34.072>
actor = 'mitya57'
assignee = 'none'
closed = True
closed_date = <Date 2012-10-04.17:59:30.043>
closer = 'pitrou'
components = ['Extension Modules', 'XML']
creation = <Date 2012-09-30.02:07:42.686>
creator = 'christian.heimes'
dependencies = []
files = ['27355', '27377', '27378', '27392', '27393']
hgrepos = []
issue_num = 16089
keywords = ['patch', '3.3regression']
message_count = 25.0
messages = ['171603', '171605', '171619', '171625', '171629', '171630', '171642', '171644', '171738', '171739', '171742', '171746', '171748', '171749', '171751', '171752', '171753', '171778', '171829', '171831', '171981', '172009', '172014', '172019', '175235']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'pitrou', 'larry', 'christian.heimes', 'ezio.melotti', 'Arfrever', 'eli.bendersky', 'skrah', 'flox', 'python-dev', 'mitya57']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue16089'
versions = ['Python 3.3', 'Python 3.4']

[tiran]

The issue was reported by Arfrever on #python-dev. The test suite of simpletal [1] was segfaulting with Python 3.3. It doesn't segfault without the _elementtree C extension.

I'm able to reproduce the issue. It may take a couple of runs with a debug build of Python but eventually Python segfaults. So far segfaults occur inside a GC collection run. Benjamin suspects the issue is related to the new GC code in _elementtree.

#0  visit_decref (op=<unknown at remote 0x7fea64fc8ca8>, data=0x0) at [Modules/gcmodule.c:361](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L361)
#1  0x00000000004bccd5 in subtract_refs (containers=<optimized out>) at [Modules/gcmodule.c:386](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L386)
#2  collect (generation=0, n_collected=0x7fffe3eedb70, n_uncollectable=0x7fffe3eedb78) at [Modules/gcmodule.c:891](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L891)
#3  0x00000000004bd7e6 in collect_with_callback (generation=0) at [Modules/gcmodule.c:1048](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L1048)
#4  collect_generations () at [Modules/gcmodule.c:1071](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L1071)
#5  _PyObject_GC_Malloc (basicsize=<optimized out>) at [Modules/gcmodule.c:1580](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L1580)
#6  0x00000000004bddcc in _PyObject_GC_Malloc (basicsize=<optimized out>) at [Modules/gcmodule.c:1567](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L1567)
#7  _PyObject_GC_New (tp=0x8322a0) at [Modules/gcmodule.c:1590](https://github.com/python/cpython/blob/main/Modules/gcmodule.c#L1590)
#8  0x0000000000548a3a in new_dict (values=0x0, keys=0xd02e60) at [Objects/dictobject.c:395](https://github.com/python/cpython/blob/main/Objects/dictobject.c#L395)
#9  _PyDict_NewPresized (minused=<optimized out>) at [Objects/dictobject.c:1044](https://github.com/python/cpython/blob/main/Objects/dictobject.c#L1044)
#10 0x000000000047a449 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at [Python/ceval.c:2245](https://github.com/python/cpython/blob/main/Python/ceval.c#L2245)
#11 0x0000000000480b03 in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7fffe3eede40, func=
    <function at remote 0x7fea653ff9e0>) at [Python/ceval.c:4150](https://github.com/python/cpython/blob/main/Python/ceval.c#L4150)

[1] http://www.owlfish.com/software/simpleTAL/downloads/SimpleTAL-5.1.tar.gz

[tiran]

The issue could be related to bpo-14007.

[skrah]

I can't reproduce this, so just a wild guess: Does the segfault still
happen if you replace Py_XDECREF() with Py_CLEAR() in xmlparser_gc_clear()?

[tiran]

No, it doesn't make a difference when I replace Py_XDECREF() with Py_CLEAR(). I've also replaced Py_(X)DECREF() in the other *_gc_clear() methods without success.

[skrah]

I'm now able to reproduce the issue with a non-debug build. As
Christian says, using Py_CLEAR() does not help (though I wonder
if it shouldn't be used anyway).

Reverting part of 20b8f0ee3d64 "fixes" the segfault, see the
attached diff. I don't know the code well enough to say if this
is a valid possibility.

[tiran]

I can confirm that Stefan's fix works. I ran the SimpleTAL test suite about hundred times in a loop without a segfault. But I don't understand why the change fixes the issue. Could the alteration just lead to another code path so the erroneous code isn't triggered?

+1 for Py_CLEAR()

[elibendersky]

I'm currently somewhat "offline" for a while (cross-continental move), but I'll do my best to try to recreate my setup to test this problem and the proposed solution.

[skrah]

Do note that the patch is somewhat cargo-cult. I don't understand yet why
it works; it may very well just silence the real problem.

[birkenfeld]

Let's make sure this gets into 3.3.1.

[pitrou]

Here is a collection of assorted small fixes for celementtree. There are probably other problems lurking (the coding style there is quite old).

I cannot say anything about the crasher until there's a simple reproducer :)

[pitrou]

More assorted celementtree cleanups.