[feat](paimon) backport paimon-cpp reader integration chain for 2.1.11-rc01 by xylaaaaa · Pull Request #61125 · apache/doris

xylaaaaa · 2026-03-08T13:24:09Z

Summary

Backport paimon-cpp reader integration chain into tmp-branch-2.1.11-rc01-paimon-cpp as a single squashed change.

Included upstream PRs:

Notes

This PR is intentionally squashed into one commit for release-branch delivery.
Local uncommitted changes in the original workspace were not touched; work was done in an isolated worktree.

Backport and integrate PR chain into tmp-branch-2.1.11-rc01-paimon-cpp: - apache#60296 - apache#60676 - apache#60711 - apache#60730 - apache#60795 - apache#60946 Not included: apache#60876

hello-stephen · 2026-03-08T13:24:14Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

Backports the paimon-cpp reader integration chain into the release branch, including thirdparty build integration, FE/BE wiring to select the C++ reader, and accompanying test coverage.

Changes:

Integrate paimon-cpp (git-based) + required thirdparty updates (RapidJSON, pugixml) and Arrow 17 patches into the thirdparty toolchain.
Add FE session/query option (enable_paimon_cpp_reader) and propagate Paimon table location + split serialization suitable for paimon-cpp.
Add BE paimon-cpp reader, predicate pushdown converter, Doris-backed paimon file system factory, plus regression/unit tests.

Reviewed changes

Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
thirdparty/vars.sh	Adds paimon-cpp + pugixml thirdparty entries; updates RapidJSON source to match paimon-cpp needs.
thirdparty/patches/paimon-cpp-buildutils-static-deps.patch	Patches paimon-cpp build to better support static-only deps and optionally reuse external Arrow.
thirdparty/patches/apache-arrow-17.0.0-paimon.patch	Applies Arrow 17 parquet/thrift-related patches needed by paimon-cpp integration.
thirdparty/paimon-cpp-cache.cmake	CMake initial cache to reuse Doris thirdparty libs (notably Arrow) when building paimon-cpp.
thirdparty/download-thirdparty.sh	Adds git-repo cloning flow for git-based thirdparty packages and patches paimon-cpp + Arrow 17.
thirdparty/build-thirdparty.sh	Builds Arrow with additional modules; adds pugixml + paimon-cpp build/install steps; adds post-build source cleanup.
regression-test/suites/external_table_p0/paimon/test_paimon_cpp_reader.groovy	Regression validating JNI vs paimon-cpp reader result parity.
gensrc/thrift/PaloInternalService.thrift	Adds thrift query option field for enabling paimon-cpp reader.
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java	Adds `enable_paimon_cpp_reader` session var and forwards it into `TQueryOptions`.
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonSource.java	Adds table location resolution for paimon-cpp.
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java	Encodes DataSplit using Paimon native serialization when paimon-cpp is enabled; passes table location.
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonUtils.java	Adds Base64 decoding fallback (URL-safe → standard) for split deserialization compatibility.
be/test/vec/exec/format/table/paimon_cpp_reader_test.cpp	Unit tests for count pushdown path and missing split error handling.
be/src/vec/exec/scan/vfile_scanner.cpp	Selects paimon-cpp reader under query option and wires predicate conversion for pushdown conjuncts.
be/src/vec/exec/format/table/paimon_predicate_converter.h	Declares converter from Doris VExpr conjuncts to paimon::Predicate.
be/src/vec/exec/format/table/paimon_predicate_converter.cpp	Implements predicate conversion (binary comparisons, IN, IS NULL, LIKE-prefix).
be/src/vec/exec/format/table/paimon_doris_file_system.h	Declares a force-link registration hook for paimon-cpp FS factory.
be/src/vec/exec/format/table/paimon_doris_file_system.cpp	Implements Doris-backed paimon::FileSystem using Doris FileFactory/FSProperties.
be/src/vec/exec/format/table/paimon_cpp_reader.h	Declares paimon-cpp GenericReader implementation.
be/src/vec/exec/format/table/paimon_cpp_reader.cpp	Implements split decode, option mapping, Arrow RecordBatch import, and COUNT pushdown via table-level row count.
be/cmake/thirdparty.cmake	Adds pugixml + paimon-cpp libs to BE thirdparty linkage when enabled.
be/CMakeLists.txt	Adds `ENABLE_PAIMON_CPP` + Arrow library selection logic and whole-archive linkage for paimon factory libs.
.github/workflows/build-thirdparty.yml	Adjusts thirdparty CI workflow (disables space maximization; lowers build parallelism).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-08T13:30:38Z

be/src/vec/exec/format/table/paimon_doris_file_system.cpp

+    std::ranges::transform(value, value.begin(),
+                           [](unsigned char c) { return static_cast<char>(std::tolower(c)); });


paimon_doris_file_system.cpp uses C++20 APIs (std::ranges::transform), but Doris BE is built with C++17 in this branch (e.g., other thirdparty builds explicitly set CPP_STANDARD=17). This will fail to compile on C++17 toolchains. Replace with C++17 equivalents (e.g., std::transform with iterators).

Suggested change

std::ranges::transform(value, value.begin(),

[](unsigned char c) { return static_cast<char>(std::tolower(c)); });

std::transform(value.begin(), value.end(), value.begin(),

[](unsigned char c) { return static_cast<char>(std::tolower(c)); });

Copilot · 2026-03-08T13:30:39Z

be/src/vec/exec/format/table/paimon_doris_file_system.cpp

+}
+
+std::string normalize_local_path(const std::string& path) {
+    if (!path.starts_with("file:")) {


C++20-only APIs are used here (e.g., std::string::starts_with). If BE is compiled as C++17 on this branch, this will not compile. Please switch to a C++17-compatible prefix check (e.g., rfind("file:", 0) == 0 / compare(0, ...) == 0).

Suggested change

if (!path.starts_with("file:")) {

if (path.compare(0, 5, "file:") != 0) {

Copilot · 2026-03-08T13:30:39Z

be/src/vec/exec/format/table/paimon_doris_file_system.cpp

+        if (type == doris::TFileType::FILE_HTTP && !options_.contains("uri") &&
+            !uri.scheme.empty()) {


C++20-only API std::map::contains is used in this condition. If the BE toolchain is C++17, this will not compile; use options_.find("uri") == options_.end() instead.

Suggested change

if (type == doris::TFileType::FILE_HTTP && !options_.contains("uri") &&

!uri.scheme.empty()) {

if (type == doris::TFileType::FILE_HTTP &&

options_.find("uri") == options_.end() && !uri.scheme.empty()) {

Copilot · 2026-03-08T13:30:39Z

.github/workflows/build-thirdparty.yml

+          #./build-thirdparty.sh -j "$(nproc)"
+          ./build-thirdparty.sh -j 2



The workflow now hard-codes the thirdparty build to ./build-thirdparty.sh -j 2 (and comments out the -j "$(nproc)" line). This can greatly slow CI and may lead to job timeouts compared to the prior configuration; consider restoring $(nproc) or making the reduced parallelism conditional (e.g., only for troubleshooting).

Suggested change

#./build-thirdparty.sh -j "$(nproc)"

./build-thirdparty.sh -j 2

./build-thirdparty.sh -j "$(nproc)"

Copilot · 2026-03-08T13:30:39Z

.github/workflows/build-thirdparty.yml

+      #- name: Maximize build space
+      #  uses: ./maximize-build-space
+      #  with:
+      #      root-reserve-mb: 4096
+      #      temp-reserve-mb: 4096
+      #      swap-size-mb: 8192
+      #      remove-dotnet: 'true'
+      #      remove-android: 'true'
+      #      remove-haskell: 'true'
+      #      remove-codeql: 'true'
+      #      remove-docker-images: 'true'


The disk-space maximization step has been commented out. If thirdparty builds still approach runner disk limits, this may cause intermittent failures; consider keeping it enabled or replacing it with an alternative space management step.

Suggested change

#- name: Maximize build space

# uses: ./maximize-build-space

# with:

# root-reserve-mb: 4096

# temp-reserve-mb: 4096

# swap-size-mb: 8192

# remove-dotnet: 'true'

# remove-android: 'true'

# remove-haskell: 'true'

# remove-codeql: 'true'

# remove-docker-images: 'true'

- name: Maximize build space

uses: ./maximize-build-space

with:

root-reserve-mb: 4096

temp-reserve-mb: 4096

swap-size-mb: 8192

remove-dotnet: 'true'

remove-android: 'true'

remove-haskell: 'true'

remove-codeql: 'true'

remove-docker-images: 'true'

[feat](paimon) backport paimon-cpp reader integration to 2.1.11-rc01

c9ca149

Backport and integrate PR chain into tmp-branch-2.1.11-rc01-paimon-cpp: - apache#60296 - apache#60676 - apache#60711 - apache#60730 - apache#60795 - apache#60946 Not included: apache#60876

Copilot AI review requested due to automatic review settings March 8, 2026 13:24

xylaaaaa requested a review from yiguolei as a code owner March 8, 2026 13:24

Copilot started reviewing on behalf of xylaaaaa March 8, 2026 13:24 View session

Copilot AI reviewed Mar 8, 2026

View reviewed changes

xylaaaaa added 2 commits March 9, 2026 14:19

fix

d3a7f25

fix: clang-format style issues

99caf5d

morningman merged commit aa776d0 into apache:tmp-branch-2.1.11-rc01-paimon-cpp Mar 10, 2026
2 of 3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat](paimon) backport paimon-cpp reader integration chain for 2.1.11-rc01#61125

[feat](paimon) backport paimon-cpp reader integration chain for 2.1.11-rc01#61125
morningman merged 3 commits intoapache:tmp-branch-2.1.11-rc01-paimon-cppfrom
xylaaaaa:backport/paimon-cppreader-2.1.11-rc01

xylaaaaa commented Mar 8, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 8, 2026

Uh oh!

Copilot AI Mar 8, 2026

Uh oh!

Copilot AI Mar 8, 2026

Uh oh!

Copilot AI Mar 8, 2026

Uh oh!

Copilot AI Mar 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		std::ranges::transform(value, value.begin(),
		[](unsigned char c) { return static_cast<char>(std::tolower(c)); });

	if (!path.starts_with("file:")) {
	if (path.compare(0, 5, "file:") != 0) {

		if (type == doris::TFileType::FILE_HTTP && !options_.contains("uri") &&
		!uri.scheme.empty()) {

		#./build-thirdparty.sh -j "$(nproc)"
		./build-thirdparty.sh -j 2

	#./build-thirdparty.sh -j "$(nproc)"
	./build-thirdparty.sh -j 2
	./build-thirdparty.sh -j "$(nproc)"

Conversation

xylaaaaa commented Mar 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Notes

Uh oh!

hello-stephen commented Mar 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xylaaaaa commented Mar 8, 2026 •

edited

Loading