Configurable error handling strategy when target partition does not exist during Stream Load #66686
Unanswered
liujunfei2230
asked this question in
A - General / Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
When using the Flink Doris Connector to write data to partitioned tables via Stream Load, if the target partition does not exist, the connector currently only has one behavior: retry and then throw a DorisBatchLoadException, which causes the entire Flink job to fail.
This is problematic in real-world streaming scenarios where:
Historical or dirty data may arrive with partition values outside the expected range (e.g., a record from last year hitting a table that only retains 30 days of partitions), causing the entire job to crash.
In multi-table sync scenarios, a single table's partition issue can bring down the entire pipeline, affecting all other tables.
Add a configurable error handling strategy for "partition does not exist" scenarios. Suggested options:
All reactions