bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type #15323

eduardo-elizondo · 2019-08-17T20:48:43Z

The instance destructor for a type is responsible for preparing
an instance for deallocation by decrementing the reference counts
of its referents.

If an instance belongs to a heap type, the type object of an instance
has its reference count decremented while for static types, which
are permanently allocated, the type object is unaffected by the
instance destructor.

Previously, the default instance destructor searched the class
hierarchy for an inherited instance destructor and, if present,
would invoke it.

Then, if the instance type is a heap type, it would decrement the
reference count of that heap type. However, this could result in the
premature destruction of a type because the inherited instance
destructor should have already decremented the reference count
of the type object.

This change avoids the premature destruction of the type object
by suppressing the decrement of its reference count when an
inherited, non-default instance destructor has been invoked.

Finally, an assertion on the Py_SIZE of a type was deleted. Heap
types have a non zero size, making this into an incorrect assertion.

https://bugs.python.org/issue37879

The instance destructor for a type is responsible for preparing an instance for deallocation by decrementing the reference counts of its referents. If an instance belongs to a heap type, the type object of an instance has its reference count decremented while for static types, which are permanently allocated, the type object is unaffected by the instance destructor. Previously, the default instance destructor searched the class hierarchy for an inherited instance destructor and, if present, would invoke it. Then, if the instance type is a heap type, it would decrement the reference count of that heap type. However, this could result in the premature destruction of a type because the inherited instance destructor should have already decremented the reference count of the type object. This change avoids the premature destruction of the type object by suppressing the decrement of its reference count when an inherited, non-default instance destructor has been invoked. Finally, an assertion on the Py_SIZE of a type was deleted. Heap types have a non zero size, making this into an incorrect assertion.

Objects/typeobject.c

jdemeyer

Apart from the one comment, the changes to typeobject.c look good. I haven't yet looked at the testcases.

Objects/typeobject.c

encukou · 2019-09-09T13:42:45Z

Modules/_testcapimodule.c

+    PyTypeObject *base = (PyTypeObject *)PyType_GetSlot(ctypesubclass, Py_tp_base);
+    Py_DECREF(ctypesubclass);
+    initproc base_init = PyType_GetSlot(base, Py_tp_init);
+    if (base_init(self, args, kwargs) < 0) {


Is this setup to get the superclass necessary?
It should be possible to call heapctype_init(self, args, kwargs) directly.

It is indeed. I actually had your suggestion initially but ultimately ended up changing it to this approach. My rationale was: To show that HeapCTypeSubclass is indeed a C extension subclass and that you can call the tp_init slot by pulling its tp_base out. To me, it seemed closer to what a __init__ would do by calling super().__init__.

That being said, I don't have any strong opinions on either. I'll keep this as is but feel free to add one more comment to change to to call heapctype_init and I will change it this time!

There are some subtle issues in doing dynamic super()-like calls from C code, but using PyState_FindModule/PyObject_GetAttrString is especially fragile.
If the elaborate code manages to set base_init to anything except heapctype_init, it's a bug. It's not the bug this is supposed to be testing; and it's a bug that's likely to segfault.

I'm actually trying to provide a better way to do this without PyState_FindModule (PEP 573), but so far, using heapctype_init is, IMO, best.

Modules/_testcapimodule.c

eduardo-elizondo · 2019-09-10T05:50:24Z

@encukou Comments have been addressed. Also, hope the core sprint is going well (if you are there) :)

- Move CAPITest additions to the end (they're testing a fairly exotic edge case, so I wouldn't make them prominent). - Have the PyModuleDef for _testcapi close to the PyInit function again (this needs a forward declaration, but keeps the module init code logically together).

The same descriptor is in the superclass.

This is mainly to make the intention clearer to readers of the code: the docstrings serve as headings.

encukou · 2019-09-10T09:44:05Z

Yup, the sprint is going well!

I have added some commits directly to this branch, rather than tell you about the nitpicks I have and let you write the code.
Could you take a look at them? Feel free to remove/change any of them (or tell me to do so) if that doesn't work for you.

eduardo-elizondo

LGTM

eduardo-elizondo

Both changes and the reasoning looks good to me

eduardo-elizondo

Nice catch!

eduardo-elizondo

Looks good

eduardo-elizondo · 2019-09-11T06:52:49Z

@encukou I went through all of you changes and they all look good! I also rebased this to master to get rid of the merge conflicts so this change should be good to go!

Given that we are in the topic of heap allocated types, I've got posixmodule to use heap allocated types only in #15892 in case you'd like to take a look 😄

miss-islington · 2019-09-11T09:26:33Z

Thanks @eduardo-elizondo for the PR, and @encukou for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2019-09-11T09:26:36Z

Sorry, @eduardo-elizondo and @encukou, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker ff023ed36ea260ab64be5895f1f1f087c798987a 3.8

…C heap type (GH-15323) The instance destructor for a type is responsible for preparing an instance for deallocation by decrementing the reference counts of its referents. If an instance belongs to a heap type, the type object of an instance has its reference count decremented while for static types, which are permanently allocated, the type object is unaffected by the instance destructor. Previously, the default instance destructor searched the class hierarchy for an inherited instance destructor and, if present, would invoke it. Then, if the instance type is a heap type, it would decrement the reference count of that heap type. However, this could result in the premature destruction of a type because the inherited instance destructor should have already decremented the reference count of the type object. This change avoids the premature destruction of the type object by suppressing the decrement of its reference count when an inherited, non-default instance destructor has been invoked. Finally, an assertion on the Py_SIZE of a type was deleted. Heap types have a non zero size, making this into an incorrect assertion. #15323. (cherry picked from commit ff023ed) Co-authored-by: Eddie Elizondo <eduardo.elizondorueda@gmail.com>

bedevere-bot · 2019-09-11T14:30:49Z

GH-15966 is a backport of this pull request to the 3.8 branch.

…C heap type (GH-15323, GH-16004) (GH-15966) The instance destructor for a type is responsible for preparing an instance for deallocation by decrementing the reference counts of its referents. If an instance belongs to a heap type, the type object of an instance has its reference count decremented while for static types, which are permanently allocated, the type object is unaffected by the instance destructor. Previously, the default instance destructor searched the class hierarchy for an inherited instance destructor and, if present, would invoke it. Then, if the instance type is a heap type, it would decrement the reference count of that heap type. However, this could result in the premature destruction of a type because the inherited instance destructor should have already decremented the reference count of the type object. This change avoids the premature destruction of the type object by suppressing the decrement of its reference count when an inherited, non-default instance destructor has been invoked. Finally, an assertion on the Py_SIZE of a type was deleted. Heap types have a non zero size, making this into an incorrect assertion. #15323. (cherry picked from commit ff023ed) Fixup: #16004. (cherry picked from commit 5e9caee) Co-authored-by: Eddie Elizondo <eduardo.elizondorueda@gmail.com>

… type (pythonGH-15323) The instance destructor for a type is responsible for preparing an instance for deallocation by decrementing the reference counts of its referents. If an instance belongs to a heap type, the type object of an instance has its reference count decremented while for static types, which are permanently allocated, the type object is unaffected by the instance destructor. Previously, the default instance destructor searched the class hierarchy for an inherited instance destructor and, if present, would invoke it. Then, if the instance type is a heap type, it would decrement the reference count of that heap type. However, this could result in the premature destruction of a type because the inherited instance destructor should have already decremented the reference count of the type object. This change avoids the premature destruction of the type object by suppressing the decrement of its reference count when an inherited, non-default instance destructor has been invoked. Finally, an assertion on the Py_SIZE of a type was deleted. Heap types have a non zero size, making this into an incorrect assertion. python#15323

the-knights-who-say-ni added the CLA signed label Aug 17, 2019

bedevere-bot added the awaiting review label Aug 17, 2019

eduardo-elizondo added 2 commits August 17, 2019 13:50

added blurb

67cef48

Fixed whitespace

d440e8a

kevinjacobs-progenity reviewed Aug 18, 2019

View reviewed changes

Objects/typeobject.c Outdated Show resolved Hide resolved

kevinjacobs-progenity reviewed Aug 18, 2019

View reviewed changes

Objects/typeobject.c Outdated Show resolved Hide resolved

Typo and indent

5bceecd

jdemeyer requested changes Aug 21, 2019

View reviewed changes

Objects/typeobject.c Show resolved Hide resolved

bedevere-bot added awaiting core review and removed awaiting review labels Aug 21, 2019

encukou reviewed Sep 9, 2019

View reviewed changes

Modules/_testcapimodule.c Outdated Show resolved Hide resolved

Fix leaks

d227ab3

encukou added 4 commits September 10, 2019 10:24

Avoid PyState_FindModule in heapctypesubclass_init

eaea7be

Remove unnecessary descriptor

6a23625

The same descriptor is in the superclass.

Add docstrings to C classes

f4258a1

This is mainly to make the intention clearer to readers of the code: the docstrings serve as headings.

eduardo-elizondo commented Sep 11, 2019

View reviewed changes

eduardo-elizondo and others added 2 commits September 11, 2019 00:31

Merge branch 'master' into fix-cheaptype-subclass

152aeaa

Remove merge artifacts

cf82442

encukou merged commit ff023ed into python:master Sep 11, 2019

bedevere-bot removed the awaiting core review label Sep 11, 2019

encukou added the needs backport to 3.8 only security fixes label Sep 11, 2019

miss-islington assigned encukou Sep 11, 2019

encukou mentioned this pull request Sep 11, 2019

[3.8] bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type (GH-15323) #15966

Merged

bedevere-bot removed the needs backport to 3.8 only security fixes label Sep 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type #15323

bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type #15323

eduardo-elizondo commented Aug 17, 2019 •

edited by bedevere-bot

Loading

jdemeyer left a comment

encukou Sep 9, 2019

eduardo-elizondo Sep 10, 2019 •

edited

Loading

encukou Sep 10, 2019

eduardo-elizondo commented Sep 10, 2019

encukou commented Sep 10, 2019

eduardo-elizondo left a comment

eduardo-elizondo left a comment

eduardo-elizondo left a comment

eduardo-elizondo left a comment

eduardo-elizondo commented Sep 11, 2019

miss-islington commented Sep 11, 2019

miss-islington commented Sep 11, 2019

bedevere-bot commented Sep 11, 2019

bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type #15323

bpo-37879: Suppress subtype_dealloc decref when base type is a C heap type #15323

Conversation

eduardo-elizondo commented Aug 17, 2019 • edited by bedevere-bot Loading

jdemeyer left a comment

Choose a reason for hiding this comment

encukou Sep 9, 2019

Choose a reason for hiding this comment

eduardo-elizondo Sep 10, 2019 • edited Loading

Choose a reason for hiding this comment

encukou Sep 10, 2019

Choose a reason for hiding this comment

eduardo-elizondo commented Sep 10, 2019

encukou commented Sep 10, 2019

eduardo-elizondo left a comment

Choose a reason for hiding this comment

eduardo-elizondo left a comment

Choose a reason for hiding this comment

eduardo-elizondo left a comment

Choose a reason for hiding this comment

eduardo-elizondo left a comment

Choose a reason for hiding this comment

eduardo-elizondo commented Sep 11, 2019

miss-islington commented Sep 11, 2019

miss-islington commented Sep 11, 2019

bedevere-bot commented Sep 11, 2019

eduardo-elizondo commented Aug 17, 2019 •

edited by bedevere-bot

Loading

eduardo-elizondo Sep 10, 2019 •

edited

Loading