[fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef - #67451
Open
morningman wants to merge 2 commits into
Open
[fix](build) Fix two arm64 BE build failures: kuromoji_build_dict link and SNII_CRC32C_X86 -Wundef#67451morningman wants to merge 2 commits into
morningman wants to merge 2 commits into
Conversation
Fixes apache#67445 `be/src/storage/index/snii/encoding/crc32c.cpp` tests `SNII_CRC32C_X86` with `#if` but only defines it on x86_64. On aarch64 the macro is undefined, and since the BE compiles with `-Wundef -Werror` the four `#if` sites fail: crc32c.cpp:105:5: error: 'SNII_CRC32C_X86' is not defined, evaluates to 0 [-Werror,-Wundef] The file is `BE_TEST`-only, so this breaks `run-be-ut.sh` on arm64 (Apple Silicon and Linux aarch64) while CI never sees it: the only aarch64 workflow builds with `-DMAKE_TEST=OFF`. Give the flag an explicit `0` on the other branch so the non-x86 build keeps only the portable slice-by-8 reference path and the hw_* seams fall back to it; the x86 branch is unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015masEhncRptQgqg3JPaRyC
Fixes apache#67448 `sh build.sh --be` fails on Apple Silicon while linking `bin/kuromoji_build_dict`: ld: fixup error (kind=arm64_b26) at '__ZN8tcmallocL14memalign_pagesEmmbb'+0x1DC from libtcmalloc.a[2](libtcmalloc_la-tcmalloc.o), B/BL out of range (displacement=-135691464, max is +/-128MB) The offline dictionary converter only calls the kuromoji builder/parser, but those return `Status`, and `Status` reaches config.cpp (-> ExecEnv), status.cpp (-> thrift/protobuf/BackendOptions) and stack_util.cpp, so the tool's link closure is effectively the whole BE and its `__TEXT` exceeds arm64's +/-128MB direct-branch reach. Apple's linker lays tcmalloc's custom `google_malloc` / `malloc_hook` sections out after `__text` and cannot insert branch islands there, so the branch from tcmalloc back to `___clang_call_terminate` at the start of `__TEXT` cannot be relaxed. The link failure also leaves the four dictionary files ungenerated, so the install step's guard fires and no `output/` is produced. This is the same failure `doris_be_test` and `benchmark_test` hit (apache#66615); the existing workaround only covers `MAKE_TEST`/`BUILD_BENCHMARK`, while the tool is built in normal builds. Apply the same treatment per target: on macOS arm64 drop tcmalloc from the tool's link line and add `gperftools_stubs.cpp` for the few gperftools symbols still referenced unconditionally. `doris_be` keeps tcmalloc; Linux link lines are unchanged. The `kuromoji_dict` ALL target and the install-time file check stay in place. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015masEhncRptQgqg3JPaRyC
morningman
requested review from
airborne12,
csun5285 and
eldenmoon
as code owners
September 2, 2026 13:10
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16922 ms |
Contributor
TPC-DS: Total hot run time: 82497 ms |
Contributor
ClickBench: Total hot run time: 14.72 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #67448, close #67445
Related PR: #64667 (introduced
kuromoji_build_dict), #66052 (introduced theSNII_CRC32C_X86test seam), #66615 (the macOS arm64 allocator workaround this PR reuses)Problem Summary:
Two BE build failures on arm64, both found while building on Apple Silicon.
1.
kuromoji_build_dictfails to link on macOS arm64 (#67448, introduced by #64667)sh build.sh --befails while linkingbin/kuromoji_build_dict:The offline dictionary converter only calls the kuromoji builder/parser, but those return
Status, andStatusreachesconfig.cpp(->ExecEnv),status.cpp(-> thrift/protobuf/BackendOptions) andstack_util.cpp, so the tool's link closure is effectively the whole BE and its__TEXTexceeds arm64's +/-128 MB direct-branch reach. Apple's linker lays tcmalloc's customgoogle_malloc/malloc_hooksections out after__textand cannot insert branch islands there, so the branch from tcmalloc back to___clang_call_terminateat the start of__TEXTcannot be relaxed. The failed link leaves the four dictionary files ungenerated, the install-time guard fires, and nooutput/is produced. The triage comment on #67448 reports the same failure on themacos-15runner of theBE UT (macOS)workflow.Trimming the tool's link line (the issue's first suggestion) is not possible without refactoring
Status, so this applies the #66615 treatment per target: on macOS arm64 the tool links against the system allocator, andgperftools_stubs.cppsupplies the few gperftools symbols that are still referenced unconditionally.doris_bekeeps tcmalloc, Linux link lines are unchanged,kuromoji_dictstays inALLand the install-time file check stays.2.
SNII_CRC32C_X86is undefined on non-x86 targets (#67445, introduced by #66052)be/src/storage/index/snii/encoding/crc32c.cpptestsSNII_CRC32C_X86with#ifbut only defines it on x86_64. The BE compiles with-Wundef -Werror, so on aarch64 the four#ifsites fail:The file is
BE_TEST-only, sorun-be-ut.shcannot build on Apple Silicon or Linux aarch64, while CI never sees it (the only aarch64 workflow builds withMAKE_TEST=OFF). Define the flag as0on the other branch: the x86 branch is unchanged, and the non-x86 build keeps only the portable slice-by-8 reference path, which thehw_*seams already fall back to.Release note
None
Check List (For Author)
Test
macOS 26.5 arm64, Homebrew clang 20.1.8, on master
72071af801d:kuromoji_build_dictin the same build directory with the pristinebe/CMakeLists.txtreproduces thearm64_b26fixup error above (__TEXTsize0x08310000, tcmalloc five times on the link line). With this changeninja -v kuromoji_build_dictlinks: notcmallocon the link line,gperftools_stubs.cpp.opresent,__TEXT0x082fc000with nogoogle_malloc/malloc_hooksections.ninja kuromoji_dictregeneratessystem.bin/matrix.bin/chardef.bin/unkdict.bin(325871 surfaces, 392126 lexicon rows, matrix 1316x1316), identical in size to the previous output.DISABLE_BUILD_UI=ON bash build.sh --beends withSuccessfully build Dorisandoutput/be/dict/kuromoji/contains all four non-empty files.crc32c.cpp:clang++ -std=c++20 -DBE_TEST -Wundef -Werror -fsyntax-onlyon arm64 reproduces the four errors before this change and passes after it;clang-formatis clean. A standalone arm64 program compiled with-DBE_TESTcomparescrc32c_slice8_extend/crc32c_hw_serial_extend/crc32c_hw3_extendagainst the bundled Google crc32c over 15 sizes x 6 alignments plus split-extend cases: 168 comparisons identical,crc32c_has_hw()reports false,"123456789"->0xE3069283.OS_MACOSX AND ARCH_ARM, so Linux link lines are byte-identical.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code
https://claude.ai/code/session_015masEhncRptQgqg3JPaRyC