ctypes array types create reference cycles

[eric-wieser]

BPO	33267
Nosy	@amauryfa, @abalkin, @meadori, @eric-wieser, @MojoVampire

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-04-12.08:45:09.451>
labels = ['ctypes']
title = 'ctypes array types create reference cycles'
updated_at = <Date 2018-04-12.22:23:40.243>
user = 'https://github.com/eric-wieser'

bugs.python.org fields:

activity = <Date 2018-04-12.22:23:40.243>
actor = 'josh.r'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2018-04-12.08:45:09.451>
creator = 'Eric.Wieser'
dependencies = []
files = []
hgrepos = []
issue_num = 33267
keywords = []
message_count = 3.0
messages = ['315216', '315217', '315236']
nosy_count = 5.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'meador.inge', 'Eric.Wieser', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue33267'
versions = ['Python 3.5']

[eric-wieser]

Discovered in https://github.com/numpy/numpy/pull/10882/files#r180813166

A reproduction:

In [1]: import ctypes

In [2]: def make_array_ctype(shape):
   ...:     import ctypes
   ...:     ct = ctypes.c_uint8
   ...:     for i in shape:
   ...:         ct = i * ct
   ...:     return ct
   ...:

# all on one line to keep ipython out of this
In [3]: gc.collect(); x = make_array_ctype((1,)); del x; gc.collect()

Using the proposed function in numpy/numpy#10891, we get a few more details:

In [4]: from numpy.testing import assert_no_gc_cycles

In [5]: assert_no_gc_cycles(make_array_ctype, (1,))
AssertionError: Reference cycles were found when calling make_array_ctype: 7 objects were collected, of which 6 are shown below:
  tuple object with id=2822255556536:
    (<class '_ctypes.Array'>,)
  PyCArrayType object with id=2822226500408:
    <class '__main__.c_ubyte_Array_1'>
  getset_descriptor object with id=2822252062256:
    <attribute '__dict__' of 'c_ubyte_Array_1' objects>
  getset_descriptor object with id=2822252062184:
    <attribute '__weakref__' of 'c_ubyte_Array_1' objects>
  tuple object with id=2822243712440:
    (<class '__main__.c_ubyte_Array_1'>,
     <class '_ctypes.Array'>,
     <class '_ctypes._CData'>,
     <class 'object'>)
  StgDict object with id=2822226211928:
    {'__dict__': <attribute '__dict__' of 'c_ubyte_Array_1' objects>,
     '__doc__': None,
     '__module__': '__main__',
     '__weakref__': <attribute '__weakref__' of 'c_ubyte_Array_1' objects>,
     '_length_': 1,
     '_type_': <class 'ctypes.c_ubyte'>}

I suppose this isn't really a bug, but it's not clear to me why a cycle needs to be created here.

[eric-wieser]

Apologies, I missed the important part of that snippet:

In [3]: gc.collect(); x = make_array_ctype((1,)); del x; gc.collect()
Out[3]: 7

[MojoVampire]

Pretty sure this is a problem with classes in general; classes are self-referencing, and using multiplication to create new ctypes array types is creating new classes.

[encukou]

Indeed, most classes create reference cycles.

>>> gc.collect(); x = type('Class', (), {}); del x; gc.collect()
5


>>> gc.collect(); x = type('Class', (), {'list_with_set': [{}]}); del x; gc.collect()
7