checkpoint timeout #5555
Comments
|
+1 |
|
i will check this. |
|
As I looked at the code below, this will generate three taskLocations with ids 200000, 400000, and 50000. PendingCheckpoint.java
line 145 if (isFullyAcknowledged()) {
line 146 LOG.debug("checkpoint is full ack!");
line 147 completableFuture.complete(toCompletedCheckpoint());
}This then causes the checkpoint to never complete and eventually times out |
|
When using JDBC to read a very large table,I found that JDBCSourceReader added a lock when reading the data, but the lock could not be retrieved from triggerBarrier in SourceFlowLifeCycle, so this task could not ack successfully. Even if I didn't need a checkpoint, I would eventually get an error if I exceeded the checkpoint timeout. Do I have to either set the timeout to the maximum value of the integer, or set the interval so large that the checkpoint fails? Is there a configuration that makes it unnecessary to checkpoint, because this kind of data itself doesn't need to be checkpointed
|
Can you set the timeout time? refer to https://seatunnel.apache.org/docs/2.3.3/seatunnel-engine/deployment#43-checkpoint-manager |
|
So is this problem just a timeout problem？ |
|
+1 |
|
I have the same question. When I synchronize a big table, I set checkpoint.timeout = a verg big number, It still get an error said CheckpointException: Checkpoint expired before completing. Please increase checkpoint timeout ... |
|
Refer to #5694. Try to use the latest code. |
I use the latest code and recompile, it still occur timeout error |
You can try to set the 'partition_column' and 'partition_num' parameters to split your table, and then set the checkpoint interval based on the processing speed of each partition. |
|
+1 |
1 similar comment
|
+1 |
|
If it is a batch task and the checkpoint timeout is not set in the task information, theoretically the checkpoint should not be triggered. Recently, I encountered this problem when running a large batch of data. The problem is not the size of the timeout setting, but whether the checkpoint should be triggered |
2.3.4 Support closing checkpoint |
Search before asking
What happened
I ran a long-running SeaTunnel task, but any time it passed the timeout node, it threw an error.
I looked around and found that the following configuration generates 3 pipelineSubtasks. But CheckpointCoordinator only goes to triggerBarrier task in startingTask every time。The remaining tasks will wait until the timeout location。
Is this because it's not done yet?
SeaTunnel Version
2.3.3
SeaTunnel Config
Running Command
Error Exception
Zeta or Flink or Spark Version
2.3.3
Java or Scala Version
No response
Screenshots
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: