​[Discussion] Potential Risks and Best Practices for Concurrent Writes to the Same Bucket from Multiple Flink Jobs​​

[lq252392]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Background

In a typical data warehouse architecture, joining multiple streams often requires separate Flink jobs (due to different sources, processing logic, or temporal reasons). This inherently leads to a scenario where multiple independent tasks (potentially on different machines) need to write to the same Paimon table, and thus, likely to the same buckets.

From the documentation and community best practices, I understand that the recommended pattern is to have a 1:1 relationship between a sink task and a bucket (i.e., sink.parallelism == num-buckets). My goal is to understand the specific technical challenges and potential risks when deviating from this pattern, and to discuss possible solutions or best practices for the multi-job writing scenario.

Research & Analysis

I have studied the relevant source code and documented my findings in this blog post: Can Paimon write to the same bucket with multiple tasks simultaneously?.

Based on my research, here are my current understandings and questions:

1. Async Compaction & Distributed Snapshots

Paimon's support for asynchronous compaction and distributed snapshot commits appears to decouple writing from compaction physically. My question:

• Does this architecture inherently allow safe concurrent commits to the same bucket from multiple tasks? (I think the answer is true)

2. sequenceNumber Behavior in Multi-Writer Scenarios

Each bucket tracks a sequenceNumber (plain long), but my analysis suggests:

• No Shared State: Since independent Flink jobs don't share the same MergeTreeWriter instance, each writer maintains its own sequence counter.
• Sorting Scope: The sequenceNumberis used ​​exclusively within SortBuffer and Compaction​​ to determine the ​​output row order. Crucially:
  • If multiple writers emit rows to the same bucket, the interleaving of their independent sequenceNumbervalues only affects the ​​physical arrangement of rows in files​​, not the logical correctness of partial update.

This leads me to hypothesize:

• If writers are inserting non-overlapping fields (e.g., different columns), could interleaved sequence numbers from different writers be harmless?
• Is the sequenceNumber irrelevant for correctness in multi-writer cases?

Questions for the Community

Could the maintainers or experienced users please help clarify:

• Are my analyses above correct? What are the other potential risks I might have missed?
• What is the recommended architecture for writing to the same Paimon table from multiple independent Flink jobs?

Thank you for your time and insights! This discussion will be incredibly helpful for designing robust pipelines with Paimon.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!