[feat](paimon) support paimon insert by jni&paimon-cpp by FayneBupt · Pull Request #62689 · apache/doris

FayneBupt · 2026-04-22T02:00:43Z

What problem does this PR solve?

Issue Number: close #56005

Related PR: #xxx

Problem Summary:

This PR implements Doris write-back support for Paimon external tables (INSERT INTO ...), completing the end-to-end write path across FE and BE, including planning, sink serialization, execution, and FE-side commit.

1) Core capability

Support writing data from Doris into Paimon external tables.
Support both writer paths:
- Native Writer (paimon-cpp): High-performance C++ native writer path (Default).
- JNI Writer: Java-based writer path for complex scenarios and compatibility.

2) FE/BE write pipeline integration

FE:
- Add Paimon insert executor flow.
- Build and send Paimon sink metadata (table identity/location/options/column names/shuffle info).
- Collect commit payload from BE and commit on FE side.
BE:
- Execute Paimon sink writing.
- Build commit payload and report back to FE.

3) Bucket/partition write behavior

Add distributed bucket shuffle support for bucketed Paimon tables.
Add compatible fallback behavior (e.g. gather/random) when required bucket keys are not resolvable.
Keep partition-related write behavior aligned with existing sink framework.

4) Limitations of Native Writer (paimon-cpp)

Currently, the high-performance Native Writer is the default path, but it has some functional limitations compared to the JNI Writer. When these scenarios are encountered, users must fallback to the JNI Writer (set enable_paimon_jni_writer = true):

Dynamic Bucket (bucket = -1): Not supported. Native Writer currently only supports fixed bucket tables.
Inline Compaction: Not supported. Native Writer operates in write-only mode without compaction capabilities during the write process.
Memory Spill: Not supported. Native Writer lacks disk spill logic when memory buffers are exhausted.
File Formats: Strictly limited to parquet for both data files and manifest files.

5) Session variables and write tunables

This PR introduces/uses Paimon-related session controls for write behavior and performance tuning.

Execution Path & Shuffle Control

enable_paimon_jni_writer (Default: false): Switch to JNI writer path instead of native paimon-cpp.
enable_paimon_distributed_bucket_shuffle (Default: true): Enable distributed bucket shuffle for Paimon tables.

Memory & Buffer Tuning

paimon_global_memory_pool_size (Default: 1073741824L / 1GB): Global memory pool size for Paimon writer.
paimon_write_buffer_size (Default: 268435456L / 256MB): Memory buffer size per writer.
enable_paimon_adaptive_buffer_size (Default: false): Enable adaptive adjustment of buffer sizes.
paimon_writer_queue_size (Default: 3): Queue size for writer tasks.
paimon_target_file_size (Default: 268435456L / 256MB): Target size for generated data files.

JNI Spill & Compaction Control (Only applicable when enable_paimon_jni_writer=true)

enable_paimon_jni_compact (Default: true): Enable inline compaction during JNI writing.
enable_paimon_jni_spill (Default: false): Enable memory spilling to disk.
paimon_spill_max_disk_size (Default: 53687091200L / 50GB): Maximum disk space allowed for spilling.
paimon_spill_sort_buffer_size (Default: 67108864L / 64MB): Buffer size for sorting during spill.
paimon_spill_sort_threshold (Default: 10): Threshold triggering sort operation.
paimon_spill_compression (Default: "zstd"): Compression codec used for spilled files.

6) Compatibility/alignment work

Align merged code with current community branch conventions and dependencies.
Resolve compile/checkstyle conflicts introduced by branch differences.
Ensure FE build passes under Maven >= 3.9 environment.

7) Current recommendation for production usage

Prefer JNI Writer in production for now: it is currently the safer path for large-scale write workloads.
Enable spill for large data volume: for big batches, it is recommended to set enable_paimon_jni_spill = true to reduce memory pressure and avoid excessive memory usage.
Native Writer roadmap dependency: Native Writer can be the default high-performance path later, but full production readiness for heavy workloads still depends on paimon-cpp upstream support for compaction and spill.

hello-stephen · 2026-04-22T02:00:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

FayneBupt · 2026-04-22T06:39:32Z

run buildall

FayneBupt · 2026-04-23T06:26:39Z

run buildall

hello-stephen · 2026-04-23T07:46:55Z

FE UT Coverage Report

Increment line coverage 35.81% (207/578) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-23T12:53:23Z

BE Regression && UT Coverage Report

Increment line coverage 6.54% (136/2081) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.25% (26299/36912)
Line Coverage	54.24% (278859/514120)
Region Coverage	51.58% (231697/449183)
Branch Coverage	52.90% (100101/189218)

hello-stephen · 2026-04-23T13:08:22Z

FE Regression Coverage Report

Increment line coverage 3.57% (45/1261) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-04-25T12:02:35Z

run buildall

hello-stephen · 2026-04-25T12:36:27Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.05% (1842/2360)
Line Coverage	64.80% (33007/50936)
Region Coverage	65.30% (16370/25070)
Branch Coverage	55.90% (8743/15640)

hello-stephen · 2026-04-25T15:37:14Z

BE UT Coverage Report

Increment line coverage 5.67% (118/2081) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.08% (20034/37744)
Line Coverage	36.57% (188869/516509)
Region Coverage	32.95% (146914/445857)
Branch Coverage	33.99% (64240/188997)

hello-stephen · 2026-04-25T16:10:49Z

BE Regression && UT Coverage Report

Increment line coverage 6.54% (136/2081) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.30% (26347/36950)
Line Coverage	54.31% (279582/514769)
Region Coverage	51.71% (232570/449761)
Branch Coverage	52.99% (100411/189503)

hello-stephen · 2026-04-25T16:20:18Z

FE Regression Coverage Report

Increment line coverage 7.76% (45/580) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-04-27T07:06:01Z

run buildall

hello-stephen · 2026-04-27T08:29:44Z

FE UT Coverage Report

Increment line coverage 51.38% (297/578) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-27T10:57:11Z

FE Regression Coverage Report

Increment line coverage 53.29% (308/578) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-27T13:45:31Z

BE UT Coverage Report

Increment line coverage 25.25% (531/2103) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.23% (20097/37752)
Line Coverage	36.68% (189493/516668)
Region Coverage	33.02% (147294/446039)
Branch Coverage	34.09% (64450/189073)

hello-stephen · 2026-04-27T13:51:15Z

BE Regression && UT Coverage Report

Increment line coverage 26.11% (549/2103) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.46% (26412/36958)
Line Coverage	54.44% (280314/514928)
Region Coverage	51.68% (232539/449943)
Branch Coverage	53.13% (100722/189579)

FayneBupt · 2026-04-28T11:46:02Z

run buildall

FayneBupt · 2026-04-30T02:10:02Z

run buildall

FayneBupt · 2026-05-06T04:15:14Z

run buildall

morningman · 2026-05-07T01:38:18Z

run buildall

Gabriel39 · 2026-05-07T01:41:17Z

/review

github-actions

I found several blocking issues in the PR. Critical checkpoints: build compatibility is not satisfied because Paimon sink symbols are referenced when ENABLE_PAIMON_CPP is disabled and the rawlog compat library is linked unconditionally; runtime correctness has risks in native writer format handling and HDFS overwrite semantics; memory-safety under allocation failure is not satisfied in the Arrow memory pool adapter. Existing review context had no prior inline threads, and there was no additional user-provided review focus beyond a complete PR review.

FayneBupt · 2026-05-07T14:29:44Z

run buildall

FayneBupt · 2026-05-07T14:43:56Z

run buildall

FayneBupt · 2026-05-08T03:13:48Z

run buildall

FayneBupt · 2026-05-12T13:58:44Z

run p0

FayneBupt · 2026-05-12T13:59:47Z

run NonConcurrent

hello-stephen · 2026-05-12T14:38:22Z

BE UT Coverage Report

Increment line coverage 27.51% (532/1934) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.38% (20228/37896)
Line Coverage	36.92% (191002/517407)
Region Coverage	33.22% (148471/446995)
Branch Coverage	34.30% (64997/189520)

hello-stephen · 2026-05-12T15:24:24Z

BE Regression && UT Coverage Report

Increment line coverage 28.02% (542/1934) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.45% (26494/37079)
Line Coverage	54.50% (280946/515523)
Region Coverage	51.88% (233888/450863)
Branch Coverage	53.13% (100956/190014)

hello-stephen · 2026-05-12T15:33:54Z

FE Regression Coverage Report

Increment line coverage 64.46% (361/560) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-05-12T16:06:35Z

BE Regression && UT Coverage Report

Increment line coverage 28.02% (542/1934) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.44% (26491/37079)
Line Coverage	54.49% (280931/515523)
Region Coverage	51.87% (233865/450863)
Branch Coverage	53.13% (100953/190014)

hello-stephen · 2026-05-12T16:07:04Z

FE Regression Coverage Report

Increment line coverage 64.46% (361/560) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-05-13T03:57:01Z

run buildall

hello-stephen · 2026-05-13T04:31:42Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.11% (1848/2366)
Line Coverage	64.85% (33223/51232)
Region Coverage	65.32% (16438/25167)
Branch Coverage	55.95% (8790/15710)

hello-stephen · 2026-05-13T05:09:13Z

FE UT Coverage Report

Increment line coverage 50.27% (278/553) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-05-13T07:46:43Z

run buildall

hello-stephen · 2026-05-13T08:17:40Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.11% (1848/2366)
Line Coverage	64.85% (33225/51232)
Region Coverage	65.34% (16445/25167)
Branch Coverage	55.88% (8779/15710)

hello-stephen · 2026-05-13T09:02:42Z

FE UT Coverage Report

Increment line coverage 50.27% (278/553) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-05-13T10:53:44Z

FE Regression Coverage Report

Increment line coverage 7.04% (39/554) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-05-13T12:25:38Z

run buildall

hello-stephen · 2026-05-13T13:34:49Z

FE UT Coverage Report

Increment line coverage 50.27% (278/553) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-05-13T16:47:13Z

FE Regression Coverage Report

Increment line coverage 8.32% (46/553) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-05-14T03:02:06Z

run buildall

hello-stephen · 2026-05-14T03:24:04Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.11% (1848/2366)
Line Coverage	64.91% (33254/51232)
Region Coverage	65.40% (16459/25167)
Branch Coverage	55.96% (8792/15710)

hello-stephen · 2026-05-14T04:08:41Z

FE UT Coverage Report

Increment line coverage 50.27% (278/553) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-05-14T06:34:20Z

FE Regression Coverage Report

Increment line coverage 8.32% (46/553) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-05-14T07:04:10Z

run buildall

hello-stephen · 2026-05-14T07:33:49Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.11% (1848/2366)
Line Coverage	64.86% (33229/51232)
Region Coverage	65.36% (16449/25167)
Branch Coverage	55.94% (8788/15710)

hello-stephen · 2026-05-14T11:05:10Z

FE Regression Coverage Report

Increment line coverage 65.10% (360/553) 🎉
Increment coverage report
Complete coverage report

FayneBupt · 2026-05-14T11:25:03Z

run p0

hello-stephen · 2026-05-14T13:35:39Z

FE Regression Coverage Report

Increment line coverage 65.10% (360/553) 🎉
Increment coverage report
Complete coverage report

FayneBupt requested a review from yiguolei as a code owner April 22, 2026 02:00

FayneBupt changed the title ~~Feature/support paimon insert 4.1~~ [feat](paimon) support paimon insert by jni&paimon-cpp Apr 22, 2026

github-actions Bot previously requested changes May 7, 2026

View reviewed changes

suxiaogang223 reviewed May 7, 2026

View reviewed changes

Comment thread be/src/exec/sink/writer/paimon/vpaimon_table_writer.cpp

suxiaogang223 reviewed May 7, 2026

View reviewed changes

Comment thread fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniWriter.java

FayneBupt force-pushed the feature/support_paimon_insert_4.1 branch from 53063d8 to 2487ebb Compare May 7, 2026 09:38

FayneBupt force-pushed the feature/support_paimon_insert_4.1 branch from 48970b4 to f1152d9 Compare May 13, 2026 07:12

FayneBupt force-pushed the feature/support_paimon_insert_4.1 branch from ebc64c5 to 217a176 Compare May 14, 2026 02:18

fix ut style

9484591

FayneBupt force-pushed the feature/support_paimon_insert_4.1 branch from 217a176 to 9484591 Compare May 14, 2026 06:49

Conversation

FayneBupt commented Apr 22, 2026

What problem does this PR solve?

1) Core capability

2) FE/BE write pipeline integration

3) Bucket/partition write behavior

4) Limitations of Native Writer (paimon-cpp)

5) Session variables and write tunables

6) Compatibility/alignment work

7) Current recommendation for production usage

Uh oh!

hello-stephen commented Apr 22, 2026

Uh oh!

FayneBupt commented Apr 22, 2026

Uh oh!

FayneBupt commented Apr 23, 2026

Uh oh!

hello-stephen commented Apr 23, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 23, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 23, 2026

FE Regression Coverage Report

Uh oh!

FayneBupt commented Apr 25, 2026

Uh oh!

hello-stephen commented Apr 25, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 25, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 25, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Apr 25, 2026

FE Regression Coverage Report

Uh oh!

FayneBupt commented Apr 27, 2026

Uh oh!

hello-stephen commented Apr 27, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 27, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Apr 27, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 27, 2026

BE Regression && UT Coverage Report

Uh oh!

FayneBupt commented Apr 28, 2026

Uh oh!

FayneBupt commented Apr 30, 2026

Uh oh!

FayneBupt commented May 6, 2026

Uh oh!

morningman commented May 7, 2026

Uh oh!

Gabriel39 commented May 7, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

FayneBupt commented May 7, 2026

Uh oh!

FayneBupt commented May 7, 2026

Uh oh!

FayneBupt commented May 8, 2026

Uh oh!