[fix](build) Strip sanitizer and platform-specific flags for OpenBLAS/FAISS#64171
Open
heguanhui wants to merge 1 commit into
Open
[fix](build) Strip sanitizer and platform-specific flags for OpenBLAS/FAISS#64171heguanhui wants to merge 1 commit into
heguanhui wants to merge 1 commit into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
…/FAISS ### What problem does this PR solve? Issue Number: close apache#64170 Problem Summary: On aarch64, building Doris BE with LSAN/ASAN/ASAN_UT/UBSAN sanitizer modes fails during CMake configuration or compilation: 1. LSAN/ASAN/ASAN_UT: The OpenBLAS getarch build-time tool inherits sanitizer flags from the parent CMake context, causing LeakSanitizer to detect memory leaks in OpenBLAS's detect() function (strdup without free on aarch64). This makes getarch exit with non-zero code and CMake configuration fails. 2. UBSAN: -mcmodel=medium is x86_64-specific and unsupported on aarch64. It causes compilation failure in both OpenBLAS getarch and Doris BE targets (e.g. ann_index.cpp). The cmake-protect layer strips it for OpenBLAS/FAISS, but the global CXX_FLAGS_UBSAN still contains it, breaking all BE targets. Also fix __lsan_ignore_object declaration: previously only enabled under ADDRESS_SANITIZER, but LEAK_SANITIZER also provides this symbol, so the empty stub was incorrectly used under LSAN mode causing link errors. The fix: - Strip sanitizer flags (-fsanitize=*, -fno-sanitize=*, -fsanitize-ignorelist=*) and -D*SANITIZER defines from CMAKE_C_FLAGS/CMAKE_CXX_FLAGS in cmake-protect CMakeLists.txt before building OpenBLAS and FAISS - Strip -mcmodel= in cmake-protect for OpenBLAS/FAISS - Move -mcmodel=medium in CXX_FLAGS_UBSAN to only apply on ARCH_AMD64 ### Release note None ### Check List (For Author) - Test: Manual test - Verified LSAN/ASAN/ASAN_UT BE UT can build and run successfully on aarch64 - Behavior changed: No - Does this need documentation: No
5d8b54e to
5ef6f7f
Compare
Contributor
|
what exactly error you have meet? |
Contributor
Author
#64170 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #64170
Related PR: #xxx
Problem Summary: On aarch64, building Doris BE with LSAN/ASAN/ASAN_UT/UBSAN sanitizer modes fails. This PR contains two commits to fix the issues.
Commit 1: Strip sanitizer and platform-specific flags for OpenBLAS/FAISS, cause run be ut failure on aarch64
LSAN/ASAN/ASAN_UT - CMake configuration failure: The OpenBLAS getarch build-time tool inherits sanitizer flags from the parent CMake context, causing LeakSanitizer to detect memory leaks in OpenBLAS's
detect()function (strdup without free on aarch64). This makes getarch exit with non-zero code and CMake configuration fails.Fix: Strip sanitizer flags (
-fsanitize=*,-fno-sanitize=*,-fsanitize-ignorelist=*) and-D*SANITIZERdefines fromCMAKE_C_FLAGS/CMAKE_CXX_FLAGSincmake-protect/CMakeLists.txtbefore building OpenBLAS and FAISS.UBSAN -
-mcmodel=mediumunsupported on aarch64:-mcmodel=mediumis x86_64-specific and hardcoded inCXX_FLAGS_UBSAN, causing compilation failure in both OpenBLAS getarch and Doris BE targets.Fix: Move
-mcmodel=mediuminCXX_FLAGS_UBSANto only apply onARCH_AMD64. Also strip-mcmodel=in cmake-protect for OpenBLAS/FAISS.TSAN - CMake configuration failure: The OpenBLAS getarch build-time tool inherits
-fsanitize=threadflag, causing getarch to produce incorrect output with undefined macros.Fix: Covered by the same sanitizer flag stripping in cmake-protect. Verified TSAN works on aarch64 after the fix.
LSAN link error:
__lsan_ignore_objectwas only enabled underADDRESS_SANITIZER, butLEAK_SANITIZERalso provides this symbol. Under LSAN mode, the empty stub was incorrectly used, causing link errors.Fix: Enable
__lsan_ignore_objectunder bothADDRESS_SANITIZERandLEAK_SANITIZERinphdr_cache.cpp.Files changed:
be/src/storage/index/ann/cmake-protect/CMakeLists.txt— Strip sanitizer flags and-mcmodel=be/CMakeLists.txt—-mcmodel=mediumonly onARCH_AMD64be/src/common/phdr_cache.cpp—__lsan_ignore_objectforLEAK_SANITIZERCommit 2: Fix UBSAN compilation failures on aarch64
After fixing
-mcmodel=mediumin commit 1, UBSAN build on aarch64 still fails due to-Wunused-macroserrors (promoted to errors by-Werror):-Wno-unused-macrossuppression not working for flex/bison generated files: Inbe/src/exprs/CMakeLists.txt,-Wno-unused-macroswas incorrectly gated behind the-Wno-unused-but-set-variablecompiler flag check. If-Wno-unused-but-set-variablewas not supported by the compiler,-Wno-unused-macroswould also be skipped, causing flex/bison generated files (wkt_lex.l.cpp,wkt_yacc.y.cpp) to fail with unused macro errors.Fix: Separate the two flag checks into independent
check_cxx_compiler_flag/ifblocks.Unused
#define PMRmacro infunction_string_misc.cpp: ThePMRmacro was defined but never used anywhere in the file.Fix: Remove the unused
#define PMRmacros.Unused
ALIGN_CACHE_LINEandPUREmacros incompiler_util.h: These macros are defined but never used anywhere in the codebase.Fix: Remove the unused macro definitions.
Files changed:
be/src/exprs/CMakeLists.txt— Separate-Wno-unused-macrosfrom-Wno-unused-but-set-variableconditionalbe/src/exprs/function/function_string_misc.cpp— Remove unused#define PMRbe/src/common/compiler_util.h— Remove unusedALIGN_CACHE_LINEandPURERelease note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)