fix: ctypes callbacks crash on macOS ARM64

[metaist]

Created by Claude Opus 4 during discussion with @metaist.

Problem

ctypes callbacks crash with "Bus error: 10" on macOS ARM64 when using Cosmopolitan Python. This affects all Python versions, not just 3.14.

Reproduction:

import ctypes

CFUNC = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_int)

def py_callback(x):
    return x * 2

cb = CFUNC(py_callback)  # Bus error here

Root Cause

Cosmopolitan's ffi_closure_alloc implementation has issues on macOS ARM64. The function is used internally by ctypes to create callable wrappers around Python functions.

Related

• Python 3.14 added a warmup call to ffi_closure_alloc during _ctypes module init (cpython#128485), which caused import ctypes to hang. This was fixed in fix: ctypes callbacks crash on macOS ARM64 #112 by removing the warmup code.
• The underlying ffi_closure_alloc issue remains and affects callback creation on all Python versions.

Impact

• Basic ctypes usage works (c_int, Structure, Array, etc.)
• Loading C libraries via ctypes does not work (expected - no dlopen)
• Creating Python callbacks for C code does not work on macOS ARM64

Workaround

Avoid using ctypes.CFUNCTYPE callbacks on macOS ARM64. Use alternative approaches like:

• Pre-compiled C extensions via cosmoext
• Pure Python implementations

Notes

• This may be a Cosmopolitan libffi bug that should be reported upstream
• Linux x86_64 behavior is untested for this specific issue

[metaist]

Investigation Update

Created by Claude Opus 4.5.

Confirmed Behavior

The issue is a hang, not a crash. The callback creation call never returns:

1. ctypes imported
2. CFUNCTYPE created  
3. Python function defined
4. About to create callback wrapper...
<hangs indefinitely>

Root Cause Analysis

• We use --disable-exec-static-tramp in our libffi build (which is documented to "prevent segfaults with closures")
• The hang occurs in ffi_closure_alloc which is called internally by ctypes when wrapping a Python function
• macOS ARM64 has strict W^X (Write XOR Execute) enforcement
• The closure allocation likely deadlocks trying to obtain executable memory

Attempted Mitigations

Our libffi build already includes:

• --disable-exec-static-tramp
• Static library only (--disable-shared --enable-static)

Upstream Issue?

This appears to be a Cosmopolitan libc issue with how ffi_closure_alloc handles memory mapping on macOS ARM64. Options:

1. Report to jart/cosmopolitan
2. Check if there's a newer Cosmopolitan version with a fix
3. Accept this as a known limitation (callbacks on macOS ARM64 don't work)

Current Status

Documented in limitations.md as a known issue. Basic ctypes functionality works; callbacks do not on macOS ARM64.

[metaist]

Deep Dive Analysis Results

Created by Claude Sonnet 4 during investigation with @metaist

Root Cause Confirmed

macOS ARM64 enforces strict W^X (Write XOR Execute) - memory pages cannot be simultaneously writable AND executable. This is a hardware-enforced security feature.

What We Tried

1. FFI_MMAP_EXEC_WRIT Path

This libffi code path creates a temp file and mmap's it twice:

• Once with PROT_READ | PROT_WRITE (for writing closure data)
• Once with PROT_READ | PROT_EXEC (for executing closure code)

Result: macOS refuses to mmap ANY file with PROT_EXEC:

Exec mmap: 0xffffffffffffffff, errno=1 (Operation not permitted)

Even files in /tmp cannot be mapped executable on macOS ARM64.

2. Direct RWX mmap

mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0)

Result: Returns EPERM on macOS ARM64.

How Apple Solves This (FFI_EXEC_TRAMPOLINE_TABLE)

libffi has a special code path for Apple platforms:

1. Pre-compiled trampoline code embedded in the library (TEXT segment, already executable)
2. Uses vm_remap() to create executable copies that jump to closure data
3. Closure data stored in a separate writable page
4. Requires Mach VM APIs: vm_allocate, vm_remap, mach_task_self()

Problem: Cosmopolitan doesn't provide Mach VM APIs.

Alternative: MAP_JIT

macOS provides MAP_JIT flag for JIT compilers:

mmap(NULL, size, PROT_READ | PROT_WRITE | PROT_EXEC, MAP_PRIVATE | MAP_ANONYMOUS | MAP_JIT, -1, 0)

Combined with pthread_jit_write_protect_np() to toggle between write and execute modes.

Problem: libffi doesn't support this pattern. The code assumes simultaneous read+write+exec or separate mappings.

Possible Solutions

Solution	Effort	Feasibility
Add MAP_JIT to libffi	High	Would require significant libffi modifications
Add Mach VM shim to Cosmopolitan	High	vm_remap may not be emulatable
Document as limitation	Low	Practical, callbacks rarely needed

Recommendation

Document ctypes callbacks as unsupported on macOS ARM64.

• ✅ ctypes calling C functions works fine
• ❌ ctypes callbacks (C calling Python) do not work
• ✅ cosmoext extensions handle this via MAP_JIT in our loader

Most Python code doesn't need ctypes callbacks. The main casualty is cffi, which uses callbacks extensively.

Impact

Package	Status	Notes
ctypes basic	✅ Works	Loading libraries, calling functions
ctypes callbacks	❌ Blocked	macOS ARM64 only
cffi	❌ Blocked	Requires callbacks
cryptography	❌ Blocked	Requires cffi
cosmoext	✅ Works	Has own MAP_JIT handling