Updating _fields_ of a derived struct type yields a bad cif

[laurialanko]

BPO	18060
Nosy	@amauryfa, @abalkin, @meadori, @websurfer5
PRs	bpo-18060: Updating _fields_ of a derived struct type yields a bad cif #13374
Files	t1.py: Test case t2.py: test case uses PyCStructUnionType_update_stgdict() but not PyCStgDict_clone()

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-05-25.19:07:26.282>
labels = ['ctypes', '3.7', '3.8', 'type-crash']
title = 'Updating _fields_ of a derived struct type yields a bad cif'
updated_at = <Date 2019-05-17.04:20:58.133>
user = 'https://bugs.python.org/laurialanko'

bugs.python.org fields:

activity = <Date 2019-05-17.04:20:58.133>
actor = 'Jeffrey.Kintscher'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['ctypes']
creation = <Date 2013-05-25.19:07:26.282>
creator = 'lauri.alanko'
dependencies = []
files = ['30369', '48332']
hgrepos = []
issue_num = 18060
keywords = ['patch']
message_count = 6.0
messages = ['189992', '228414', '342505', '342520', '342532', '342676']
nosy_count = 5.0
nosy_names = ['amaury.forgeotdarc', 'belopolsky', 'meador.inge', 'lauri.alanko', 'Jeffrey.Kintscher']
pr_nums = ['13374']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue18060'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.7', 'Python 3.8']

Linked PRs

• [3.12] gh-62260: Fix ctypes.Structure subclassing with multiple layers (GH-13374) #113623
• [3.11] gh-62260: Fix ctypes.Structure subclassing with multiple layers (GH-13374) #113624

[laurialanko]

In Modules/_ctypes/stgdict.c:567 there is a suspicious line:

stgdict->length = len;      /* ADD ffi_ofs? */

That is, the length field of the stgdict is set to the number of fields in the immediate Structure class, and the number of fields in the parent class (ffi_ofs) is questionably left out. This is wrong.

The length field is used in PyCStgDict_clone to copy the ffi_type descriptors for struct elements to a derived struct type. If length is short, not all element types are copied, and the resulting array is not NULL-terminated.

So the problem manifests when you inherit from a structure type, update the _fields_ of the inherited type, and then inherit again from the updated type. Even then everything might seem normal, since the elements array is actually not used very much.

However, attached is a test case that segfaults at least with debug builds on ARM with the VFP ABI. The non-null-terminated element type array is traversed to find if the structure can be passed in floating point registers, eventually resulting in dereferencing 0xfbfbfbfb.

The test program should print out pi. To avoid the hassle of a separate C component, the program abuses the standard atan2 function by pretending it takes a struct of two doubles instead of two separate double parameters. This does not make a difference to the ABI.

Fixing the bug is trivial. Just change the line to:

stgdict->length = ffi_ofs + len;

[BreamoreBoy]

A one line change is given in msg189992 so is this correct or isn't it?

[websurfer5]

It still behaves as described in the 3.7 and master branches. The proposed fix works. I will submit a pull request.

[websurfer5]

While the fix works as advertised, it breaks a similar existing test case:

======================================================================
ERROR: test_positional_args (ctypes.test.test_structures.StructureTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/jeff/sandbox/src/python3.7/cpython/Lib/ctypes/test/test_structures.py", line 390, in test_positional_args
    z = Z(1, 2, 3, 4, 5, 6)
IndexError: list index out of range

---

Below is the relevant test code:

        class W(Structure):
            _fields_ = [("a", c_int), ("b", c_int)]
        class X(W):
            _fields_ = [("c", c_int)]
        class Y(X):
            pass
        class Z(Y):
            _fields_ = [("d", c_int), ("e", c_int), ("f", c_int)]

        z = Z(1, 2, 3, 4, 5, 6)

It similar but slightly different from the provided test case:

        libm = CDLL(find_library('m'))

        class Base(Structure):
            _fields_ = [('y', c_double),
                        ('x', c_double)]

        class Mid(Base):
            pass

        Mid._fields_ = []

        class Vector(Mid): pass

        libm.atan2.argtypes = [Vector]
        libm.atan2.restype = c_double

        arg = Vector(y=0.0, x=-1.0)

I will do some more digging.

[websurfer5]

The current behavior,

stgdict->length = len;

sets the number of elements in the class excluding its base classes. The new behavior of

stgdict->length = ffi_ofs + len;

sets the total number of elements in both the class and its base classes.

This works as intended as long as the child classes do not add any more elements.

[websurfer5]

The t1.py test case calls both PyCStructUnionType_update_stgdict() and PyCStgDict_clone(), both of which are broken. The test case in t2.py is simpler than t1.py in that it only calls PyCStructUnionType_update_stgdict().

PyCStructUnionType_update_stgdict() gets called whenever _fields_ gets assigned a new list of element tuples. The function is supposed to copy any inherited element pointers in parent classes and append the new elements. The element pointers array in each child class is supposed to be cumulative (i.e. parent class element pointers plus child class element pointers).

_fields_ is represented by a StgDictObject structure, and the relevant member variables are 'ffi_type_pointer.elements', 'len', and 'size'. 'ffi_type_pointer.elements' is an array of ffi_type pointers padded with a NULL pointer at the end. 'size' is the number of bytes in the array excluding the padding. 'len' is the number of elements in the current class (i.e. excluding the elements in parent classes).

PyCStructUnionType_update_stgdict() allocates a new 'ffi_type_pointer.elements' array by adding 1 to the sum of 'len' and the 'len' member of the parent class, then multiplying by sizeof(ffi_type *). This works just fine when there is a single parent class, but breaks when there are multiple levels of inheritance. For example:

   class Base(Structure):
      _fields_ = [('y', c_double),
                  ('x', c_double)]

    class Mid(Base):
        _fields_ = []

    class Vector(Mid):
        _fields_ = []

PyCStructUnionType_update_stgdict() gets called for each of the three _fileds_ assignments. Assuming a pointer size of 8 bytes, Base has these values:

    ffi_type_pointer.elements = array of 3 pointers (x, y, and NULL padding).
    len = 2
    size = 16 (i.e. 2 * sizeof(ffi_type *))

Mid has these values:

    ffi_type_pointer.elements = array of 3 pointers (x, y, and NULL padding).
    len = 0
    size = 16 (i.e. 2 * sizeof(ffi_type *))

Vector has these values:

    ffi_type_pointer.elements = array of 1 pointer (x)
    len = 0
    size = 16 (i.e. 2 * sizeof(ffi_type *))

Vector's 'len' and 'size' are correct, but 'ffi_type_pointer.elements' contains one element instead of three. Vector should have:

    ffi_type_pointer.elements = array of 3 pointers (x, y, and NULL padding).
    len = 0
    size = 16 (i.e. 2 * sizeof(ffi_type *))

'ffi_type_pointer.elements' got truncated because PyCStructUnionType_update_stgdict() uses the parent class's 'len' field to determine the size of the new array to allocate. As can be seen, Mid's 'len' is zero, so a new array with one element gets allocated and copied (0 element pointers plus a trailing NULL pointer for padding). Notice that Vector's 'size' is correct because the value is calculated as Mid's 'size' plus zero (for zero elements being added in the child class).

Similarly, PyCStgDict_clone() has the same problem because it also uses the similar calculations based on 'len' to determine the new 'ffi_type_pointer.elements' array size that gets allocated and copied.

The solution proposed by lauri.alanko effectively redefines the 'len' member variable to be the total number of elements defined in the inheritance chain for _fields_. While this does fix the allocation/copying issue, it breaks other code that expects the 'len' variables in the parent and child classes to be distinct values instead of cumulative. For example (from StructureTestCase.test_positional_args() in Lib/ctypes/test/test_structures.py),

    class W(Structure):
        _fields_ = [("a", c_int), ("b", c_int)]
    class X(W):
        _fields_ = [("c", c_int)]
    class Y(X):
        pass
    class Z(Y):
        _fields_ = [("d", c_int), ("e", c_int), ("f", c_int)]

    z = Z(1, 2, 3, 4, 5, 6)

will throw an exception because Z's 'len' will be 6 instead of the expected 3.

A better solution is to use 'size' to allocate and copy 'ffi_type_pointer.elements' since its value is already properly calculated and propagated through inheritance.

[serhiy-storchaka]

The length field was used in two incompatible meanings. In PyCStgDict_clone() it was the total number of items in ffi_type_pointer.elements (excluding the trailing null). In PyCStructUnionType_update_stgdict() and _init_pos_args() it was the the number of items added in this class, not including inherited items. There are two ways to resolve this:

1. Calculate the total number in PyCStgDict_clone() by traversing recursively base classes.
2. Keep the total number in length and change PyCStructUnionType_update_stgdict() and _init_pos_args().

I choose the second option as simpler in long term.