tp_version_tag is not unique when test runs with -R : #89077
Comments
|
tp_version_tag is supposed to be unique for different class objects. Under normal circumstances, everything works properly: def good():
class C:
def __init__(self): # Just to force `tp_version_tag` to update
pass
cls_id = hex(id(C))
tp_version_tag_before = C.v # v is tp_version_tag of C, exposed to Python
x = C() # tp_new requires a _PyType_Lookup for `__init__`, updating `tp_version_tag`
tp_version_tag_after = C.v
print(f'C ID: {cls_id}, before: {tp_version_tag_before} after: {tp_version_tag_after}')
for _ in range(100):
good()Result: However, wrapping in a unittest and run under -R : suddenly changes things: class BadTest(unittest.TestCase):
def test_bad(self):
class C:
def __init__(self):
pass
cls_id = hex(id(C))
tp_version_tag_before = C.v
x = C()
tp_version_tag_after = C.v
print(f'C ID: {cls_id}, before: {tp_version_tag_before} after: {tp_version_tag_after}')Result: Somehow the class is occasionally occupying the same address, and tp_version_tag didn't update properly. tp_version_tag being unique is an important invariant required for LOAD_ATTR and LOAD_METHOD specialization. I bumped into this problem after LOAD_METHOD specialization kept failing magically in test_descr. I think this is related to bpo-43636 and bpo-43452, but I ran out of time to bisect after spending a day chasing this down. I'll try to bisect soon. |
Fidget-Spinner
added
3.11
|
Thanks for putting in the effort to find this. I think the first step to fixing this is to formalize the semantics of |
|
@mark, agreed about properly defining the semantics. For the current bug: I've narrowed it down to a corner case in how the -R tests work:
One possible fix: we can tell sys._clear_type_cache to not reset next_version_tag to 0 (it can use another function, since PyType_ClearCache is part of public C API and shouldn't be modified). This seems like a bug in libregrtest, the type method cache and tp_version_tag aren't at fault for this specific scenario. I'm submitting a PR soon. |
|
PyType_ClearCache() is documented as: Modifying it to do what it is documented to do fixes this bug :) |
|
Thanks Mark and Victor for the patch and reviews! This issue also affects 3.10, but IMO we don't need to backport this as we don't have other code relying on this invariant in that version, so it won't affect anything there. If you feel otherwise, please let me know. The fix and tests have landed, I'm closing this issue. |
|
The opcode cache for load attribute in 3.10 uses the tp_version_tag so unless I am missing something we can fall into the same problem, no? |
|
@pablo yup the 3.10 opcache used tp_version_tag. But it also does identity (pointer) comparison of type/class object https://github.com/python/cpython/blob/3.10/Python/ceval.c#L3432. Which is why it doesn't fail. I created this issue because we don't do type pointer comparisons anymore in 3.11's new specialization infrastructure, and rely only on tp_version_tag and dict keys version. |
Good point! Even if I wrote that code I forgot about all the guards that were needed for correctness. I now remember all the weird cases we discovered there :) |
|
I just realised I'm slightly wrong about identity checks -- there is a very very small chance where if the type object occupies the same address and the attribute is in a dynamically allocated __dict__ (and not some static slot), we can trick the checks and a segfault may occur for LOAD_ATTR using the old opcache in 3.10. I've tried running tests for 30 repetitions, but I cannot get the segfault to appear. @pablo I'll leave it up to you if you feel it's backport worthy. I don't think people use sys._clear_type_cache or PyType_ClearCache normally so the chances of this causing any crash is astronomically small. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: