fix: ensure extensions link correctly with libneug on Linux#5
Merged
fix: ensure extensions link correctly with libneug on Linux#5
Conversation
When ENABLE_GCOV=ON (push to main), src/ targets are compiled with
--coverage but extension targets are not. This mismatch causes the
extension shared library to fail resolving symbols from the
coverage-instrumented libneug.so at runtime via dlopen.
Also remove the redundant ${ARROW_LIB} link from the json extension,
since it is already transitively provided through neug's PUBLIC
dependency.
Made-with: Cursor
longbinlai
changed the title
longbinlai
force-pushed
the
fix-gcov-extension
branch
3 times, most recently
from
ec810d2 to
0a1ad92
Compare
When the host process (e.g. Python) loads libneug.so with RTLD_LOCAL,
neug symbols stay in a local scope and are invisible to extensions
loaded via dlopen, even though the extensions list libneug.so in
DT_NEEDED. This causes "undefined symbol" errors for neug symbols
like InvalidArgumentException.
Fix by re-opening libneug.so with RTLD_NOLOAD | RTLD_GLOBAL in the
load_extension() path. This promotes the already-loaded instance to
global visibility without reloading, allowing extensions to resolve
neug symbols. The promotion is done once, only when extensions are
actually loaded.
Also:
- Add --coverage flags for extension builds when ENABLE_GCOV is active
- Remove redundant ${ARROW_LIB} from json extension (avoids ODR with
statically-linked Arrow already in libneug.so)
- Add diagnostic CI step for extension linkage debugging
Made-with: Cursor
longbinlai
force-pushed
the
fix-gcov-extension
branch
from
0a1ad92 to
9f701ae
Compare
shirly121
approved these changes
Mar 8, 2026
Louyk14
pushed a commit
that referenced
this pull request
Mar 12, 2026
Introduce pybind and nexg_python_bind
BingqingLyu
added a commit
to BingqingLyu/neug
that referenced
this pull request
Apr 8, 2026
- Fix empty endpoint override being ignored (comment #2) - Modified getOptionWithEnv to respect explicitly set empty values - Allows ENDPOINT_OVERRIDE="" to force default AWS S3 endpoint - Fix HTTP ReadAt returning incorrect byte count (comment alibaba#3) - Changed ReadRange to return actual bytes read (Result<int64_t>) - Updated ReadAt to return actual bytes instead of requested nbytes - Handle zero-length reads and HTTP 416 responses - Use arrow::SliceBuffer for correct buffer sizing - Fix HTTP timeout parsing uncaught exceptions (comment alibaba#4) - Added try-catch around std::stoi for timeout values - Added std::exception handler in OpenInputFile - Fix s3_extension_test missing curl link library (comment alibaba#5) - Added CURL_LIBRARIES and CURL_INCLUDE_DIRS to test target
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the
undefined symbolerror when loading the JSON extension viadlopenat runtime.Root cause
When Python imports
neug_py_bind, it loadslibneug.sowithRTLD_LOCALby default, keeping all neug symbols private to the importing module. When the JSON extension is later loaded viadlopen()(triggered byLOAD JSON), the dynamic linker cannot resolve neug symbols (e.g.,InvalidArgumentException), resulting inundefined symbolerrors.Changes
tools/python_bind/neug/__init__.py: SetRTLD_GLOBALbefore importingneug_py_bindso thatlibneug.sosymbols are globally visible for subsequentdlopencalls. This is the standard pattern used by PyTorch, TensorFlow, and other Python packages that load plugins dynamically.extension/CMakeLists.txt: Add--coveragecompile/link flags to extension builds whenENABLE_GCOVis active, ensuring consistent instrumentation with the core library (fixes the push-to-main GCOV failure).extension/json/CMakeLists.txt: Remove redundant${ARROW_LIB}from the json extension link. Arrow is already statically embedded inlibneug.soand propagated transitively. Duplicating it causes ODR violations.Test plan