Reference cycle when importing ctypes

[poq]

BPO	12142
Nosy	@terryjreedy, @amauryfa, @meadori, @eryksun, @iritkatriel
Files	ctypes-leak.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2011-05-21.22:39:48.199>
labels = ['ctypes', '3.11', 'performance']
title = 'Reference cycle when importing ctypes'
updated_at = <Date 2021-12-09.13:37:57.706>
user = 'https://bugs.python.org/poq'

bugs.python.org fields:

activity = <Date 2021-12-09.13:37:57.706>
actor = 'eryksun'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2011-05-21.22:39:48.199>
creator = 'poq'
dependencies = []
files = ['24413']
hgrepos = []
issue_num = 12142
keywords = ['patch']
message_count = 11.0
messages = ['136485', '137148', '137371', '137379', '140098', '152552', '152661', '164258', '164262', '408111', '408121']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'amaury.forgeotdarc', 'meador.inge', 'python-dev', 'poq', 'vladris', 'eryksun', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue12142'
versions = ['Python 3.11']

[poq]

When importing ctypes after gc.set_debug(gc.DEBUG_LEAK), the garbage collector finds a 'c_int_Array_3' class and some related objects.

The class is created in ctypes/_endian.py:
_array_type = type(c_int * 3)

It seems that this could be avoided with:
_array_type = type(Array)

Of course, I realize this is not a bug because normally it will just get collected. It is just an extremely minor annoyance because this is currently the only thing still found by DEBUG_LEAK for my program ;)

[terryjreedy]

If you are able to rebuild Python, have you tried running the ctypes test after rebuilding with this change? And, does the test cover the internal uses of _array_type?

[poq]

Tests succeed with this change.

There is only one use of _array_type, which is in the same module. This use is presumably tested, because the test fails if I change the line to _array_type = type(Structure).

In fact, everything must behave exactly the same after this change, because the two values are identical:

>>> from ctypes import *
>>> type(c_int * 3) is type(Array)
True

[terryjreedy]

Thank you for the test and explanation. There currently is no specific cytpes maintainer. But from what you have said, I might feel comfortable enough applying this, if no one else does, when I have the necessary setup on a new machine.

[vladris]

I ran full test suit after making the _array_type = type(Array) change and everything passes.

I also took a look at this and found additional leak. gc shows this as garbage:

[(<class '_ctypes._SimpleCData'>,), <class 'ctypes.c_longdouble'>, <attribute '_
_dict__' of 'c_longdouble' objects>, <attribute '__weakref__' of 'c_longdouble'
objects>, (<class 'ctypes.c_longdouble'>, <class '_ctypes._SimpleCData'>, <class
'_ctypes._CData'>, <class 'object'>), {'__dict__': <attribute '__dict__' of 'c_
longdouble' objects>, '_type_': 'g', '__module__': 'ctypes', '__weakref__': <att
ribute '__weakref__' of 'c_longdouble' objects>, '__doc__': None}]

This is all caused by these lines in ctypes __init__.py:

class c_longdouble(_SimpleCData):
    _type_ = "g"
if sizeof(c_longdouble) == sizeof(c_double):
    c_longdouble = c_double

For me sizeof(c_longdouble) == sizeof(c_double) (I believe MS compiler always does this) but when we assign c_longdouble = c_double, there is a leak. I removed the alias lines:

if sizeof(c_longdouble) == sizeof(c_double):
    c_longdouble = c_double

And the leak was gone. Looks like changing c_longdouble after declaring it causes a leak. Below for similar aliasing of longlong types, we have this:

if _calcsize("l") == _calcsize("q"):
    # if long and long long have the same size, make c_longlong an alias for c_long
    c_longlong = c_long
    c_ulonglong = c_ulong
else:
    class c_longlong(_SimpleCData):
        _type_ = "q"
    _check_size(c_longlong)

    class c_ulonglong(_SimpleCData):
        _type_ = "Q"

This avoids declaring c_longlong and c_ulonglong as class if not needed to. The problem is _calcsize("g") causes an error because "g" is used as long double througout ctypes but _calcsize is function from _struct.c, where "g" (long double) is not defined. Not sure why it isn't...

So in short:
As far as I can tell _array_type = type(Array) doesn't break anything
Looks like we have another leak in ctypes (which isn't a big deal)
We have elegant fix for the leak once _struct.c will support long double

[poq]

I've attached a patch for the _array_type change.

The long double fix is probably dependent on PEP-3118 (bpo-3132).

[python-dev]

New changeset 205da7a19a78 by Meador Inge in branch '3.2':
Issue bpo-12142: Fixed reference cycle when importing ctypes
http://hg.python.org/cpython/rev/205da7a19a78

New changeset b228d9da8bd3 by Meador Inge in branch 'default':
Issue bpo-12142: Fixed reference cycle when importing ctypes
http://hg.python.org/cpython/rev/b228d9da8bd3

New changeset 7cdbf627f958 by Meador Inge in branch '2.7':
Issue bpo-12142: Fixed reference cycle when importing ctypes
http://hg.python.org/cpython/rev/7cdbf627f958

[amauryfa]

Meador, can we close this issue?

[meadori]

Meador, can we close this issue?

I wanted to keep it open until the 'long double' problem is fixed as well.

[iritkatriel]

Looks like the long double issue is still there in 3.11

>>> import gc
>>> gc.set_debug(gc.DEBUG_LEAK)
>>> import ctypes
gc: collectable <function 0x0000026417BBE200>
gc: collectable <tuple 0x0000026417BE0040>
gc: collectable <dict 0x0000026417BC56C0>
gc: collectable <type 0x0000026417AD9840>
gc: collectable <tuple 0x0000026417BD6540>
gc: collectable <getset_descriptor 0x0000026417BD6580>
gc: collectable <getset_descriptor 0x0000026417BD65C0>
gc: collectable <tuple 0x0000026417BE05E0>
gc: collectable <_ctypes.PyCSimpleType 0x0000026417AE4E10>
gc: collectable <tuple 0x0000026417BD96C0>
gc: collectable <getset_descriptor 0x0000026417BE9080>
gc: collectable <getset_descriptor 0x0000026417BE90C0>
gc: collectable <StgDict 0x00000264178A5490>
>>> gc.garbage
[<function _C._m at 0x0000026417BBE200>, (<class 'object'>,), {'__module__': 'types', '_m': <function _C._m at 0x0000026417BBE200>, '__dict__': <attribute '__dict__' of '_C' objects>, '__weakref__': <attribute '__weakref__' of '_C' objects>, '__doc__': None}, <class 'types._C'>, (<class 'types._C'>, <class 'object'>), <attribute '__dict__' of '_C' objects>, <attribute '__weakref__' of '_C' objects>, (<class '_ctypes._SimpleCData'>,), <class 'ctypes.c_longdouble'>, (<class 'ctypes.c_longdouble'>, <class '_ctypes._SimpleCData'>, <class '_ctypes._CData'>, <class 'object'>), <attribute '__dict__' of 'c_longdouble' objects>, <attribute '__weakref__' of 'c_longdouble' objects>, {'__module__': 'ctypes', '_type_': 'g', '__dict__': <attribute '__dict__' of 'c_longdouble' objects>, '__weakref__': <attribute '__weakref__' of 'c_longdouble' objects>, '__doc__': None}]

[eryksun]

The _ctypes extension module could have a dict that maps each format code to its (size, alignment), based on formattable. Then direct size comparisons wouldn't be limited to types defined by the struct module, and it wouldn't be necessary to create c_longdouble just to check its size and throw it away.

[furkanonder]

Looks like the long double issue is still there in 3.11

>>> import gc
>>> gc.set_debug(gc.DEBUG_LEAK)
>>> import ctypes
gc: collectable <function 0x0000026417BBE200>
gc: collectable <tuple 0x0000026417BE0040>
gc: collectable <dict 0x0000026417BC56C0>
gc: collectable <type 0x0000026417AD9840>
gc: collectable <tuple 0x0000026417BD6540>
gc: collectable <getset_descriptor 0x0000026417BD6580>
gc: collectable <getset_descriptor 0x0000026417BD65C0>
gc: collectable <tuple 0x0000026417BE05E0>
gc: collectable <_ctypes.PyCSimpleType 0x0000026417AE4E10>
gc: collectable <tuple 0x0000026417BD96C0>
gc: collectable <getset_descriptor 0x0000026417BE9080>
gc: collectable <getset_descriptor 0x0000026417BE90C0>
gc: collectable <StgDict 0x00000264178A5490>
>>> gc.garbage
[<function _C._m at 0x0000026417BBE200>, (<class 'object'>,), {'__module__': 'types', '_m': <function _C._m at 0x0000026417BBE200>, '__dict__': <attribute '__dict__' of '_C' objects>, '__weakref__': <attribute '__weakref__' of '_C' objects>, '__doc__': None}, <class 'types._C'>, (<class 'types._C'>, <class 'object'>), <attribute '__dict__' of '_C' objects>, <attribute '__weakref__' of '_C' objects>, (<class '_ctypes._SimpleCData'>,), <class 'ctypes.c_longdouble'>, (<class 'ctypes.c_longdouble'>, <class '_ctypes._SimpleCData'>, <class '_ctypes._CData'>, <class 'object'>), <attribute '__dict__' of 'c_longdouble' objects>, <attribute '__weakref__' of 'c_longdouble' objects>, {'__module__': 'ctypes', '_type_': 'g', '__dict__': <attribute '__dict__' of 'c_longdouble' objects>, '__weakref__': <attribute '__weakref__' of 'c_longdouble' objects>, '__doc__': None}]

The patch has been applied and the problem seems to be solved in 3.13.

╰─$ ./python                
Python 3.13.0a0 (heads/main:2b7027d, Jun  2 2023, 23:42:56) [GCC 13.1.1 20230429] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gc
>>> gc.set_debug(gc.DEBUG_LEAK)
>>> import ctypes
>>> gc.garbage
[]
>>>