feat(build): support Linux aarch64 builds

[SteNicholas]

Purpose

Linked issue: close #171

Paimon C++ only built and was validated on x86_64: the architecture-specific build logic keyed on PAIMON_CPU_FLAG and PAIMON_ARMV8_ARCH, neither of which was ever defined, so the armv8 tuning branch was dead code and Arm64 got no -march= flag at all. This change resolves the target architecture once and keys every architecture-specific decision on it. Running the suite on ubuntu-24.04-arm then exposed two classes of target-dependent behavior — plain-char signedness, which the AArch64 Linux ABI decides differently, and undefined float-to-integer conversion — plus a gap in how a failure is reported.

Build system and CI:

• cmake_modules/TargetArchitecture.cmake derives PAIMON_TARGET_PROCESSOR and PAIMON_TARGET_CPU_FAMILY from CMAKE_SYSTEM_PROCESSOR, with no compiler probe so cmake -P can test it. An unrecognized processor gets no tuning flags rather than a configure error.
• PAIMON_AARCH64_MARCH selects the Arm64 -march= value, defaulting to armv8-a; an explicitly empty value passes no -march flag at all. A fallback covers add_subdirectory() consumers, for which the option does not exist.
• The dead SIMD build logic is removed. The SSE4.2 CRC32C kernel computed Castagnoli, not the zlib CRC-32 persisted in the SST block trailer and the B-tree global index, and it hung off the never-defined PAIMON_SIMD_LEVEL, so no build ever selected it — no on-disk value changes. The ppc AltiVec probe and the never-read PAIMON_HAVE_NEON / PAIMON_HAVE_ARMV8_CRC / PAIMON_HAVE_ARMV8_CRYPTO definitions go with it.
• Lumina is prebuilt for linux-x86_64 only: configuring it elsewhere now fails with an explicit error, and CI disables it off x86_64.
• build_and_package.sh names the artifact after the platform, e.g. output/paimon-cpp-linux-x86_64.tar.gz. --platform overrides the label alone (cross compilation is not wired up end to end); --print-name prints the resolved name and exits.
• CI runs the same matrix on both architectures — Debug and Release, GCC and Clang, ASan/UBSan and TSan — with clang-tidy the one exception since it analyses the source. Job names now carry the architecture (gcc-debug-x86_64, gcc-debug-aarch64, …), a script-tests job is added, and .asf.yaml is renamed and extended in lockstep so all jobs stay required checks. Release verification gains an aarch64 entry in its own workflow.

Defects the aarch64 run exposed:

• BinaryString::NumBytesForFirstByte classified a UTF-8 leading byte by its sign. With char unsigned on aarch64, every byte was taken for single-byte ASCII, so NumChars, Substring and IndexOf counted bytes instead of characters — NumChars("Paimon中文社区") returned 18 rather than 10. The byte is now classified as uint8_t, branch for branch equivalent to the signed form on x86-64, so x86-64 results are unchanged.
• ToUpperCase and ToLowerCase passed bytes to toupper / tolower as plain char — undefined behavior for bytes at or above 0x80 where char is signed, x86-64 included. They now go through unsigned char, on the fast path and in the CppTo*Case fallbacks, and a stray pre-loop tolower write in ToUpperCase is removed. Intended results do not change.
• Min and max aggregation compared TINYINT variants as plain char, so on aarch64 min(-20, 10) returned 10. They compare the signed value now, as does VariantValueToString, which printed 236 for -20.
• The sanitizer build did not link on aarch64: UBSan's overflow check on 128-bit multiplications calls __muloti4, which libgcc does not provide and Clang inlines on x86-64 but not on aarch64. The Clang UBSan build now links compiler-rt's builtins archive.
• Converting an out-of-range or non-finite float to an integer was undefined behavior, in NumericPrimitiveCastExecutor and in the Arrow kernel it delegated to. JavaFloatingToIntegerCast now defines it as Java does — NaN becomes 0, out-of-range saturates at the int32 bounds (int64 for BIGINT), then narrows to the target width by keeping the low bits — on both the literal and the array path, so stats and data stay consistent. This changes results on x86-64, where those conversions previously produced whatever the hardware did.
• asan_symbolize.py truncated the failure report: it decoded stdin with surrogateescape but re-opened stdout strictly, so a byte that is not valid UTF-8 broke the pipe — which is why the first aarch64 run reported only a single failure. Both streams now use surrogateescape.

Tests

New script-mode tests, run by the script-tests CI job without a toolchain or a configured build tree:

• ci/scripts/test_cmake_modules.sh — processor-string mapping in TargetArchitecture.cmake, and the PAIMON_AARCH64_MARCH default, override and add_subdirectory() fallback.
• ci/scripts/test_packaging_args.sh — build_and_package.sh argument parsing, the default platform label, --platform validation, --print-name and the resulting archive name.
• ci/scripts/test_asan_symbolize.sh — invalid UTF-8 bytes round-trip unchanged and symbolization continues past them; each case was verified to fail against the unfixed script.

Updated unit tests (target unittest):

• crc32c_test.cpp pins the checksum to zlib CRC-32 reference values rather than to whichever kernel was compiled in.
• cast_executor_test.cpp covers all eight float/double to integer pairs with the same twelve inputs and expected Java results on both the literal and the array path, including MAX/LOWEST/±INFINITY/NaN, plus the empty-array, sliced-offset and null edges of the conversion that replaced Arrow's kernel.
• field_min_max_agg_test.cpp and data_define_test.cpp cover the negative TINYINT; two utils tests now read it back through int8_t.
• binary_string_test.cpp enables the four-byte UTF-8 case as u8"\U0001F919" (it was disabled as an ill-formed surrogate pair), adds the same code point to TestSubstring and TestIndexOf, and adds invalid UTF-8 bytes passing through case conversion unchanged. The existing multi-byte tests were the ones failing on aarch64; they are unchanged and pass there now.

The gcc-debug-aarch64 and gcc-release-aarch64 jobs run the full unit and integration suite on ubuntu-24.04-arm, the end-to-end verification that the library builds and passes on aarch64 in both configurations.

API and Format

No change to the public API under include/, and no change to the storage format: the removed CRC32C kernel was never selected by any build, and BinaryString::NumBytesForFirstByte keeps its char parameter and its x86-64 results, returning those same values on aarch64 now.

One behavior change: casting a float or double whose truncated value does not fit an integer type now follows Java Paimon on every architecture. It was undefined behavior before, so x86-64 results for those inputs change — CAST(MAX_FLOAT AS TINYINT) yields -1 instead of 0, CAST(300.9 AS TINYINT) yields 44, and CAST(NaN AS INT) yields 0 instead of INT_MIN. Values that already fit are unaffected.

Build-facing changes for downstream users: a new PAIMON_AARCH64_MARCH option, the released archive is now named paimon-cpp-<platform>.tar.gz instead of paimon-cpp.tar.gz (with --platform / --print-name added to build_and_package.sh), and configuring with lumina enabled off linux-x86_64 now fails explicitly instead of failing later at link time.

Documentation

Yes. docs/source/building.rst gains a supported platform matrix and documents PAIMON_AARCH64_MARCH and the platform-labelled packaging. docs/source/user_guide/read.rst updates the type change support matrix: float/double to integer casts are now well defined and Java-consistent on every supported architecture. docs/code-style.md gains rules against relying on the signedness of plain char and on undefined float-to-integer conversions, pointing at JavaFloatingToIntegerCast as the policy Paimon applies.

Generative AI tooling

Generated-by: Claude Opus 5 (1M context)

[zjw1111]

Thanks for taking on the aarch64 support work — the architecture handling is much cleaner after this. One question about the ppc handling that this refactor carries over.

[lszskye]

+1

[zjw1111]

+1

[JingsongLi]

+1

[zjw1111]

@u70b3 Sorry for not pinging you during the review stage of this PR — that was an oversight on my side.

This PR has already been merged. Whenever you have some time, could you take a look and check whether the current fix meets your use case? If anything is still missing or does not work as expected on your side, please let us know and we will follow up.

[u70b3]

@zjw1111 No worries, and thanks for driving the aarch64 port!

I took it for a proper spin on my local aarch64 machine (a 128-core Kunpeng-920): replicated the full CI aarch64 matrix locally — gcc debug/release, clang debug/release, ASan+UBSan, and TSan. Everything builds cleanly and the entire test suite passes (40/40). The port works well for my use case. 🎉

While validating, I also ran a deeper ARM portability sweep and found a few suspected/confirmed issues that #181 didn't cover, so I went ahead and fixed them locally:

• a real double-checked-locking race in Singleton<T> — hidden by x86 TSO, but TSan-confirmed on aarch64 with torn reads observed under a 128-thread stress litmus
• a TSan-confirmed race on IOHook's mode_ (fault-injection/test-only impact)
• the plain-char INT8 pattern that survived feat(build): support Linux aarch64 builds #181 in field_sum_agg.cpp (latent, same bug class as the min/max fix)
• a strict-aliasing UB in SerializationUtils::DeserializeBinaryRow (latent)
• arch-divergent double→int conversions in cache_manager.h / sst_file_writer.cpp (latent, pathological configs only)

Fixes plus deterministic TSan regression tests are in #203 — details and the red→green evidence are in the PR description. Happy to split the PR or adjust anything if you'd prefer a different shape!