Skip to content

branch-4.1: [feature](paimon) Paimon write support via JNI#65868

Open
suxiaogang223 wants to merge 3 commits into
apache:branch-4.1from
suxiaogang223:codex/backport-paimon-jni-write-4.1
Open

branch-4.1: [feature](paimon) Paimon write support via JNI#65868
suxiaogang223 wants to merge 3 commits into
apache:branch-4.1from
suxiaogang223:codex/backport-paimon-jni-write-4.1

Conversation

@suxiaogang223

@suxiaogang223 suxiaogang223 commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

Implement Paimon table write support (INSERT INTO / INSERT OVERWRITE) for Doris via JNI, bridging to the Paimon Java SDK. The architecture uses a single-writer model (analogous to Iceberg) with partition and bucket routing delegated to the Paimon SDK.

Architecture

VPaimonTableWriter (single AsyncResultWriter)
  └── IPaimonWriteBackend
        ├── JniPaimonWriteBackend  ← Java SDK via JNI
        └── FfiPaimonWriteBackend  ← stub for future Rust backend

Changes

Layer Description
Thrift TPaimonTableSink, TPaimonCommitMessage
BE IPaimonWriteBackend / IPaimonWriter / IPaimonCommitter interfaces; JNI backend; FFI stub; VPaimonTableWriter; Pipeline Operator
FE Nereids planner integration; PaimonTransaction; PaimonInsertExecutor
Java PaimonJniWriter (Arrow IPC → Paimon SDK)

Not Included

  • Paimon DDL (CREATE TABLE) — tables created externally
  • Compaction, Lookup/FullCompaction Changelog Producer
  • Rust FFI backend

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65381

Problem Summary: Doris 4.1 can read Paimon external tables but does not have a write path. This backport adds INSERT INTO and INSERT OVERWRITE through a JNI bridge to the Paimon Java SDK, including planner and transaction integration, type conversion, partitioned writes, commit and abort handling, outcome reconciliation, and Doris-managed spill paths. The backport preserves the branch-4.1 JDBC-specific sink and FileSystemProvider APIs while adapting the Paimon sink plan copies to the branch-4.1 Nereids interfaces.

### Release note

Add Paimon external table write support through JNI.

### Check List (For Author)

- Test: Unit Test
    - `./build.sh --fe --be-java-extensions`
    - `./run-fe-ut.sh --run org.apache.doris.datasource.paimon.PaimonTransactionTest,org.apache.doris.paimon.PaimonJniWriterTest,org.apache.doris.nereids.parser.ParseInsertPartitionSpecTest`
    - `JAVA_TOOL_OPTIONS="--add-opens=java.base/java.nio=ALL-UNNAMED" ./run-fe-ut.sh --run org.apache.doris.paimon.PaimonJniWriterTest`
- Behavior changed: Yes, Paimon external tables support INSERT INTO and INSERT OVERWRITE
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
suxiaogang223 marked this pull request as ready for review July 22, 2026 03:35
@suxiaogang223
suxiaogang223 requested a review from yiguolei as a code owner July 22, 2026 03:35
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@suxiaogang223 suxiaogang223 changed the title [feature](paimon) Add Paimon write support via JNI (branch-4.1) branch-4.1: [feature](paimon) Paimon write support via JNI Jul 22, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 3.62% (16/442) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.88% (28598/39238)
Line Coverage 56.38% (309604/549162)
Region Coverage 53.61% (258791/482765)
Branch Coverage 54.57% (113606/208184)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants