Crash in _ctypes_alloc_callback

[ogre]

BPO	14354
Nosy	@brettcannon, @ronaldoussoren, @amauryfa, @pitrou, @ned-deily, @meadori, @willingc
Superseder	bpo-16575: ctypes: unions as arguments
Files	testctypes.py: Test case for crash in _ctypes_alloc_callback on Mac 64 bit

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-06-23.04:57:15.964>
created_at = <Date 2012-03-17.20:55:43.738>
labels = ['ctypes', 'type-crash']
title = 'Crash in _ctypes_alloc_callback'
updated_at = <Date 2016-06-23.09:30:48.986>
user = 'https://bugs.python.org/ogre'

bugs.python.org fields:

activity = <Date 2016-06-23.09:30:48.986>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2016-06-23.04:57:15.964>
closer = 'willingc'
components = ['ctypes']
creation = <Date 2012-03-17.20:55:43.738>
creator = 'ogre'
dependencies = []
files = ['24919']
hgrepos = []
issue_num = 14354
keywords = []
message_count = 10.0
messages = ['156205', '156210', '156218', '156219', '156236', '156242', '156243', '156430', '192425', '269106']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'amaury.forgeotdarc', 'pitrou', 'ned.deily', 'meador.inge', 'ogre', 'willingc']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '16575'
type = 'crash'
url = 'https://bugs.python.org/issue14354'
versions = ['Python 2.7', 'Python 3.2']

[ogre]

I have reproduced this crash in Apple's default 2.7.1 python, and in 2.7.3 built from source myself. But only in release mode. If I rebuild 2.7.3 in debug, the crash goes away.

The attached file reproduces the issue, which has to do with a union containing multiple structs being used as a value for a callback argument. I've stripped it down as much as I can. Removing any fields from either the union or the Dice struct will cause this to no longer crash.

The original source of this is libtcod's python wrapper library, which seems to work fine on platforms other than Mac, though it had a lot of other problems on 64-bit systems before I got to this one. This issue may also be more 64-bit specific than Mac specific.

This is the callstack from Apple's python:

#0 0x00000001010c7712 in _ctypes_alloc_callback ()
#1 0x00000001010c4a1c in PyCData_AtAddress ()
#2 0x0000000100050afa in icu::DigitList::getDouble ()
#3 0x000000010000bd32 in PyObject_Call ()
#4 0x000000010008bf63 in ubrk_swap ()
#5 0x000000010008ecd8 in triedict_swap ()
#6 0x000000010008ed4d in triedict_swap ()
#7 0x00000001000a608f in ucnv_openStandardNames ()
#8 0x00000001000a614f in ucnv_openStandardNames ()
#9 0x00000001000a72a2 in ucnv_getUnicodeSet ()
#10 0x00000001000b72af in ucnv_getDisplayName ()
#11 0x0000000100000e88 in ?? ()

[ogre]

I just built python 3.2.2 from source, and reproduced the issue there as well. Same location. Here's the slightly more informative stack trace from my release-with-symbols 3.2.2 build:

#0 _ctypes_alloc_callback (callable=0x7fff5fbfef20, converters=0x100000003, restype=0x7fff5fbfef20, flags=1606414112) at callbacks.c:432
#1 0x00000001010c5395 in PyCFuncPtr_new (type=0x7fff5fbfefc0, args=0x101010580, kwds=0x7fff5fbfefc0) at _ctypes.c:3372
#2 0x000000010004986c in type_call (type=0x1006ca940, args=0x101041b90, kwds=0x0) at typeobject.c:676
#3 0x0000000100008605 in PyObject_Call (func=0x1006ca940, arg=0x101041b90, kw=0x0) at abstract.c:2149
#4 0x000000010008afcc in do_call [inlined] () at /Users/ogre/src/Python-3.2.2/Python/ceval.c:4141
#5 0x000000010008afcc in PyEval_EvalFrameEx (f=0x7fff5fbff1c0, throwflag=1606414784) at ceval.c:3944
#6 0x000000010009131b in PyEval_EvalCodeEx (_co=0x7fff5fbff260, globals=0x0, locals=0x1066c8800000101, args=0x7fff5fbff260, argcount=1606414944, kws=0x7fff5fbff260, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at ceval.c:3350
#7 0x000000010009139f in PyEval_EvalCode (co=0x101066828, globals=0x101066c88, locals=0x0) at ceval.c:767
#8 0x00000001000b0a21 in run_mod [inlined] () at /Users/ogre/src/Python-3.2.2/Python/pythonrun.c:1783
#9 0x00000001000b0a21 in PyRun_FileExFlags (fp=0x7fff7c677ee0, filename=0x101066828 "\002", start=0, globals=0x101066c88, locals=0x0, closeit=1, flags=0x7fff5fbff420) at pythonrun.c:1740
#10 0x00000001000b2992 in PyRun_SimpleFileExFlags (fp=0x7fff7c677ee0, filename=0x10104ac20 "testctypes.py", closeit=1606415200, flags=0x7fff5fbff360) at pythonrun.c:1265
#11 0x00000001000c45af in run_file [inlined] () at /Users/ogre/src/Python-3.2.2/Modules/main.c:297
#12 0x00000001000c45af in Py_Main (argc=1606415440, argv=0x7fff5fbff450) at main.c:692
#13 0x0000000100001522 in main (argc=17197096, argv=0x100609fe0) at python.c:59

converters doesn't seem to be pointing at valid data.

(gdb) p *converters
$2 = {
ob_refcnt = 3302829852670,
ob_type = 0xe0000000200
}

But, being an optimized build, it's hard to say (especially for me, having never debugged python itself before) if that's the real data or some optimizer-mangled version.

[ned-deily]

This is likely related to bpo-13370 which documents test_ctypes failures when using either of the llvm-based compilers (clang or llvm-gcc) supplied with recent versions of Xcode 4. test_ctypes and your test both do not fail when Python is compiled with the standard (non-llvm) gcc-4.2 supplied in Xcode 3.2.6 (for OS X 10.6).

[ogre]

Thanks for that. This does seem to be the case. I rebuilt with CC=gcc-4.2 and my short sample and the full library I took it from both work fine.

[amauryfa]

I also noticed that the script crashes on Debian 64bit as well, versions 2.6 and 2.7.3rc2 at least, optimized builds.
So it's not a clang issue after all.

[amauryfa]

There is a out-of-bounds error in Modules/_ctypes/libffi/src/x86/ffi64.c:
at line 225, classes[i + pos] can go outside the allocated memory for classes (MAX_CLASSES=4).
This code is not prepared to received "structures" with a small size (<32bytes) but where individual elements total more than 32bytes.

libffi support for unions is weak; at least ctypes should not use FFI_TYPE_STRUCT for unions.

[amauryfa]

Not a Mac issue.

[ogre]

It's maybe not directly relevant to fixing this, but I worked around it on the project where this came up by redefining dice as (c_int * 4) and col as (c_uint8 * 3) in the union, then using ctypes.cast to get those as pointers to the actual struct. That seems to work just fine for now.

[ronaldoussoren]

The example uses unions, which libffi doesn't really support (see for example <https://github.com/atgreen/libffi/issues/33\>).

There is an issue in the python tracker about union support in ctypes: bpo-16575.

FWIW. I can still reproduce the issue with x86_64 on OSX with both the 2.7 and default branches, while i386 does not crash.

I intend to close this as a duplicate of bpo-16575 as that better explains the underlying problem.

[willingc]

As per Ronald's last comment "close this as a duplicate of bpo-16575 as that better explains the underlying problem".

Closing this dated and duplicate issue.