[fix](be) Fix macOS Apple Silicon (arm64) compilation and build issues#63474
Open
yx-keith wants to merge 4 commits into
Open
[fix](be) Fix macOS Apple Silicon (arm64) compilation and build issues#63474yx-keith wants to merge 4 commits into
yx-keith wants to merge 4 commits into
Conversation
765f029 to
1453067
Compare
6765b6e to
966d176
Compare
This commit collects all changes required to compile and run the BE unit tests on macOS 15+ / Apple Silicon (arm64) with Apple Clang 17. - run-be-ut.sh: auto-detect Apple Clang 17 and Homebrew JDK-17 on macOS - be/CMakeLists.txt: set DISABLE_ANN compile definition *before* add_subdirectory(storage/index/ann) so ann_index sees the flag; always build ann_index (production code links against it regardless of FAISS availability) - be/src/storage/CMakeLists.txt, be/src/exprs/CMakeLists.txt: always link ann_index - be/test/CMakeLists.txt: always exclude ANN test files from the monolithic doris_be_test target - be/src/storage/index/ann/CMakeLists.txt: guard OpenMP / faiss link with if(NOT DISABLE_ANN) - be/src/storage/index/ann/cmake-protect/CMakeLists.txt: strip ASAN/UBSAN flags from OpenBLAS sub-build (prevents getarch hang on macOS); disable OpenMP in OpenBLAS on Apple Silicon; force ARMV8 target - be/src/common/factory_creator.h: use std::shared_ptr(new T) instead of std::make_shared on Apple to avoid libc++ compressed_pair bug - be/src/common/multi_version.h: use atomic_shared_ptr wrapper - be/src/core/binary_cast.hpp: add missing #include <bit> - be/src/core/value/vdatetime_value.h: guard non-const copy ctor with #ifndef __APPLE__ (Apple Clang 17 rejects it for trivially-copyable) - be/src/exec/common/memory.cpp: disambiguate std::max literal types - be/src/load/routine_load/kinesis_conf.cpp: explicit int64_t cast for Aws::Utils::DateTime constructor - be/src/macos_patches/__ranges/lazy_split_view.h: libc++ patch for std::ranges::lazy_split_view used in tests - be/src/storage/index/ann/ann_index_writer.cpp, be/src/storage/index/ann/ann_index_reader.cpp: guard faiss includes and FaissVectorIndex usage with #ifndef DISABLE_ANN - be/src/storage/index/ann/faiss_ann_index.cpp, be/src/storage/index/ann/faiss_ann_index.h: wrap entire file body with #ifndef DISABLE_ANN - be/src/exprs/function/array/function_array_distance.h: provide inline FAISS stubs under #else DISABLE_ANN so L1/L2/IP distance classes compile without FAISS headers - be/src/storage/segment/segment_writer.h: move _is_mow() and _is_mow_with_cluster_key() inline definitions from the .cpp to the .h; macOS linker dead-strips out-of-line inline symbols, causing TestSegmentWriter link failures - be/src/storage/segment/segment_writer.cpp: remove the now-redundant inline definitions - be/test/udf/python/python_env_test.cpp: replace sighandler_t (Linux glibc extension) with void (*)(int) - be/test/storage/segment/inverted_index_reader_test.cpp: local copy for vector<bool> element to avoid proxy-reference UB in template - be/test/storage/index/ann/*.cpp: guard faiss / HNSW includes with #ifndef DISABLE_ANN - Multiple test files: add LL suffix to large integer literals, replace std::ranges::iota_view loops, add missing #include <thread>, and disambiguate int64_t literal types to satisfy Apple Clang 17
…clang-format - Ignore vendored libc++ patch dir be/src/macos_patches in license check (carries the LLVM Apache-2.0-WITH-LLVM-exception header, not the ASF one) - Reformat compaction_permit_limiter_test and inverted_index_reader_test to satisfy clang-format
966d176 to
202bc08
Compare
… runner The macos-15 GitHub runner is now Apple Silicon (arm64): - JAVA_HOME_17_X64 is empty on arm64, so set JAVA_HOME from the Homebrew openjdk@17 install (and install openjdk@17 instead of @11); env.sh requires JDK-17. - Download the darwin-arm64 prebuilt thirdparty (matching uname -m) instead of the hardcoded x86_64 tarball, which both mismatches the runner arch and lacks libarrow_dataset.a.
202bc08 to
4403abe
Compare
Followup to apache#63557 which renamed COW::assume_mutable to assert_mutable but missed 20 test files. Linux CI may pass these files via PCH or build-order luck; macOS clang catches them as unresolved member references and breaks the BE UT build.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit collects all changes required to compile and run the BE unit tests on macOS 15+ / Apple Silicon (arm64) with Apple Clang 17.
Build system (CMake / run-be-ut.sh)
Source portability fixes
Test portability fixes
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)