-
Notifications
You must be signed in to change notification settings - Fork 76
Python wheel should export DuckDB C API symbols (or provide a path to them) #404
Description
What happens?
Prior to 1.4.1, the DuckDB Python wheel exported all C API symbols on every platform. Starting with 1.4.1, duckdb-python#81 restricted exports to only PyInit__duckdb and duckdb_adbc_init via linker flags in CMakeLists.txt L83-L110.
On macOS, -exported_symbol is a hard allowlist — all C API symbols are gone. On Linux, --export-dynamic-symbol is additive, so all symbols remain visible as a side effect. This means the C API works on Linux today but is broken on macOS — and could break on Linux too if a version script is added in the future.
| Version | macOS C API symbols | Linux C API symbols |
|---|---|---|
| 1.3.x | ~6,300 ✅ | ~6,300 ✅ |
| 1.4.0 | ~6,300 ✅ | ~5,900 ✅ |
| 1.4.1+ | 0 ❌ | ~5,900 ✅ (by accident) |
| 1.5.x | 0 ❌ | ~6,300 ✅ (by accident) |
This is a breaking change for projects that load the C API from the Python wheel.
Use case
numbduck wraps 160 DuckDB C API functions for use inside numba @njit compiled code. It loads the shared library from the installed duckdb Python package and resolves C API symbols at runtime. This worked on all platforms through duckdb 1.4.0, but breaks on macOS starting with 1.4.1.
The standalone libduckdb downloads have all symbols, but requiring a separate install (brew install duckdb) alongside pip install duckdb is a friction point — users expect the Python package to be self-contained.
Possible solutions
- Export all
duckdb_*C API symbols from the Python wheel on all platforms. The symbols are already compiled into the binary — they're just hidden by the linker flags. - Add a
ctypes-friendly entry point — e.g. a function that returns a pointer to a struct of C API function pointers (similar to howduckdb_adbc_initworks for ADBC). - Bundle
libduckdb.dylib/.soalongside the Python extension in the wheel, soctypes.CDLLcan load it directly. - Document the limitation — if the C API is intentionally not part of the Python wheel's public interface, document that C API consumers should use the standalone
libduckdbdownloads.
Option 1 is the smallest change — just remove the linker flags. The PR #81 commit message says these are "the only two [symbols] that are ever needed," but that assumes all consumers go through Python or ADBC.
Environment
- duckdb 1.4.1 through 1.5.1 (macOS broken; Linux works by accident)
- macOS ARM64 and x86_64
- Verified with
nm -gD/nm -gon installed wheel.sofiles