-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for Flink's SpeculativeExecution in batch execution mode - Backport of PR #10548 #10776
Conversation
backport from 1.19 with some changes
cc @pvary for review. |
The 1st point is fine. Thanks, Peter |
We didn't have |
@rodmeneses: IIRC we faced an issue in the tests when we were introducing Flink 1.19. This was solved by adding the If my recollection above is correct, then we have 2 tasks ahead of us:
@rodmeneses: Could you please confirm? Thanks, |
that is correct @pvary. We discussed about having a more thorough cleaning logic for all of FlinkSQL related unit tests, but I didn't have the time nor I wanted to keep adding more complexity on my previous PR. |
@pvary Can you clarify what do you mean by the "inconsistencies" here? |
Differences would have been a better word. Based on the discussion above: @venkata91: Could you please remove the Thanks, |
cf6e44d
to
3c4c34a
Compare
Okay removed the |
@pvary Gentle ping |
@venkata91: Could you please check the failure? |
@pvary I looked at it earlier and it seems to be unrelated. Test failure is in |
@pvary Btw, I merged the |
Merged to main. |
Summary
Add support for Flink's Speculative Execution in batch execution mode
Details
Backport from 1.19 to 1.18 required some manual conflict resolution only in the tests. They are as follows:
getRuntimeContext().getTaskInfo().getIndexOfThisSubtask()
andgetRuntimeContext().getTaskInfo().getAttemptNumber()
usegetRuntimeContext().getIndexOfThisSubtask()
andgetRuntimeContext().getAttemptNumber()
dropDatabase()
helper method inTestBase
which required addingDEFAULT_CATALOG_NAME
as a constant inFlinkCatalogFactory
.Backport from 1.18 to 1.17 went through cleanly.
Testing
Added an integration test to verify the expected speculative execution behavior with IcebergSource