ctypes's c_uint64 is effectively an alias to uint32 due to misuse of struct.calcsize()

[Nodraak]

Bug report

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

3.10

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]

A clear and concise description of the bug:

Hi all,

ctypes's c_uint64 is effectively an alias to uint32 due to misuse of struct.calcsize() in ctypes's code.

Here is a minimal, reproducible example:

import ctypes, struct

bs = bytearray.fromhex("000000000006a440")  # "435264" as big endian 64 bits unsigned integer

val1 = struct.unpack_from("!Q", bs)
print("unpack_from Q (hardcoded): %s" % (val1))

val2 = struct.unpack_from("!" + ctypes.c_uint64._type_, bs)
print("unpack_from %s (ctypes.c_uint64): %s" % (ctypes.c_uint64._type_, val2))

The above will prints:

unpack_from Q (hardcoded): 435264  // unpack_from() is fine
unpack_from L (ctypes.c_uint64): (0,)  // but not ctypes.c_uint64._type_...

While I would expect:

unpack_from Q (hardcoded): 435264
unpack_from Q (ctypes.c_uint64): 435264  // Correct unpack

Symptom: ctypes.c_uint64._type_ is "L", but should be "Q"

Suggested fix: in /usr/lib/python3.10/ctypes/__init__.py:206, replace if _calcsize("l") == _calcsize("q"): by if _calcsize("=l") == _calcsize("=q"):

Rationale: without "=", the size including padding is returned: 32 bits is aligned
on 64 bits. That's wrong, we want the real size, without padding.

---

Version:

• Python: Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
• uname: Linux LAPTOP-ADRIEN 5.19.0-43-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon May 22 13:39:36 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Quick links:

• https://docs.python.org/3/library/struct.html#format-characters
• https://docs.python.org/3/library/struct.html#byte-order-size-and-alignment
• https://docs.python.org/3/library/ctypes.html#ctypes.c_ulonglong
• https://github.com/python/cpython/blob/main/Lib/ctypes/__init__.py

Linked PRs

• gh-108356: ctypes: fix ctypes's c_uint64's _type_ #108359

[FrozenBob]

This doesn't look like a bug. ctypes.c_uint64._type_ is supposed to be interpreted with native size, not standard size. The ! in your test and the = in your commit both force standard size.

[FrozenBob]

As you're on Linux, native size for long is in fact 64 bits. struct.calcsize isn't including any padding.

Try actually constructing an instance of ctypes.c_uint64 and storing and retrieving a value that wouldn't fit in a 32-bit int:

import ctypes
x = ctypes.c_uint64()
x.value = 2**40
print(x.value)
print(x.value == 2**40)

and compare the results to what you get with ctypes.c_uint32. The results should show that ctypes.c_uint64 is really 64-bit.

[ambv]

I agree with @FrozenBob. This is not a bug. ctypes exposes your system's native data sizes and endianness. Compile this on your machine to see that long is in fact long long:

#include <stdio.h>

int main() {
    printf("sizeof(ll) = %lu\n", sizeof(long long));
    printf("sizeof(l) = %lu\n", sizeof(long));
    printf("sizeof(i) = %lu\n", sizeof(int));
    return 0;
}

On a 64-bit system the results will be 8, 8, 4. This is the same as:

>>> struct.calcsize(ctypes.c_longlong._type_)
8
>>> struct.calcsize(ctypes.c_long._type_)
8
>>> struct.calcsize(ctypes.c_int._type_)
4

[Nodraak]

ctypes.c_uint64._type_ is L, which, according to https://docs.python.org/3/library/struct.html#format-characters is 4 bytes. I think it should be Q.

I expect the following to print 12345, not 0:

serialized = bytearray.fromhex("%016x" % 12345)
deserialized = struct.unpack_from(
    "!" + ctypes.c_uint64._type_,
    serialized,
)
print(deserialized)

This clearly looks like a bug to me. Am I missing something?

Edit: I want to unpack a 64 bits integer from raw bytes (0000000000003039) into a Python int (12345). How can I do that? I would expect the above example to work, but it does not.

[FrozenBob]

That table shows standard sizes, not native sizes. Native sizes are platform-dependent.

If you want to unpack a 64-bit unsigned integer from a network-order bytearray into a standard Python int, you should not use ctypes at all, especially not ctypes.c_uint64._type_. ctypes.c_uint64._type_ is an undocumented implementation detail - _type_ is only documented for ctypes.Array and ctypes._Pointer.

Instead, you should just use

deserialized = struct.unpack('!Q', serialized)

(and remember that struct.unpack will give you a 1-item tuple, not the int directly.)

[Nodraak]

I understand that long has a different size for native vs standard, and that ctypes.c_uint64._type_ is undocumented / a private member, but I find it very surprising that one can not get the correct struct's Format Characters like with the other types.

If you say it is the expected behavior, that's fine. Then I will ask about my specific use case on stackoverflow.

Thank you for the explanation