ctypes pointer not always keeping target alive

[arigo]

BPO	2123
Nosy	@arigo, @theller, @bitdancer
Files	x.py: failing test or bogus test? y.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/theller'
closed_at = None
created_at = <Date 2008-02-15.14:24:58.157>
labels = ['extension-modules', 'type-bug']
title = 'ctypes pointer not always keeping target alive'
updated_at = <Date 2013-07-10.10:14:10.589>
user = 'https://github.com/arigo'

bugs.python.org fields:

activity = <Date 2013-07-10.10:14:10.589>
actor = 'christian.heimes'
assignee = 'theller'
closed = False
closed_date = None
closer = None
components = ['Extension Modules']
creation = <Date 2008-02-15.14:24:58.157>
creator = 'arigo'
dependencies = []
files = ['9436', '9438']
hgrepos = []
issue_num = 2123
keywords = []
message_count = 5.0
messages = ['62429', '62436', '62453', '84117', '84218']
nosy_count = 3.0
nosy_names = ['arigo', 'theller', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'languishing'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue2123'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

Linked PRs

• gh-46376: Return existing pointer when possible in ctypes #107131
• [3.12] gh-46376: Return existing pointer when possible in ctypes (GH-107131) #107487
• [3.11] gh-46376: Return existing pointer when possible in ctypes (GH-107131) #107488
• [3.11] Revert "gh-46376: Return existing pointer when possible in ctypes (GH-107131) (GH-107488)" #108412
• gh-46376: add (failing) tests for ctypes/pointer/cast/set_contents #108519
• Revert "gh-46376: Return existing pointer when possible in ctypes (#1… #108688
• [3.12] gh-46376: Revert "Return existing pointer when possible in ctypes (GH-107131) (GH-107487)" #108864

[arigo]

It's hard to tell for sure, given the lack of precise definition, but I
believe that the attached piece of code "should" work. What it does is
make p1 point to c_long(20). So ctypes should probably keep the
c_long(20) alive as long as p1 is alive (and not further modified).
This test shows that the c_long(20) gets freed instead, making the
p1.contents reference garbage.

[theller]

May I ask: do you have a real use case for this, or is it a carefully
constructed example?

Of course I take all the blame for not defining/documenting this
stuff. My current view is this:

Python code C code
======================= ================

ptr = POINTER(c_long)()         int *ptr = NULL;
x = c_long(42)                  int x = 42;

ptr.contents = x                ptr = &x;

a = ptr[0]                      int a = *ptr;
b = ptr[n]                      int b = ptr[n];

Assigning to .contents changes 'where the pointer points to'.
__setitem__ changes the pointed to memory location; __getitem__
retrieves the pointed to memory location.

Having said that, it is no longer clear to me what reading the
.contents attribute should mean. Would making the .contents attribute
write-only help - is it impossible to construct this 'bug' without
assigning to .contents?

[arigo]

We're finding such bugs because we are trying to reimplement ctypes in PyPy.

I guess your last question was "is it impossible to construct this 'bug'
without *reading* .contents?". The answer is that it doesn't change
much; you can replace all the reads from .contents with reads from [0],
and still see the issue. See attached y.py, where I've put the
equivalent C code in comments.

[bitdancer]

Thomas, do you accept this as a bug or should the issue be closed as
invalid?

[theller]

I accept this as a bug; however I don't have time now to work on it.

[ambv]

This looks like it works as expected at least as of Python 3.7 where c_long(20) is indeed preserved and p1.contents removes 20 as expected.

[code-of-kpp]

Original examples would pass, but only because of gc delay.

Here's fun way to demonstrate that the problem still exists (and can be converted to a test):

import ctypes
import weakref

p1 = ctypes.pointer(ctypes.c_long(10))
w = weakref.WeakValueDictionary()
ctypes.pointer(p1).contents.contents = w.setdefault(
    1, ctypes.c_long(-1),
)
assert p1.contents.value == -1
assert 1 in w

[ambv]

This ancient issue is now fixed. Thanks, Konstantin! ✨ 🍰 ✨

[code-of-kpp]

This should be re-opened, as fix in 3.11 was reverted and the one in 3.12/3.13/main isn't working always

[vstinner]

This should be re-opened, as fix in 3.11 was reverted and the one in 3.12/3.13/main isn't working always

Right, see issue #107940 which is a Python 3.12 regression.

In the 3.12 branch, I confirm that the commit 54aaaad introduced the regression: issue #46376 of PR #107487 (backport of PR #107131).

[code-of-kpp]

Please, take a look at
#108519