-
Notifications
You must be signed in to change notification settings - Fork 13k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-28899][table-planner] Fix LOOKUP hint with retry option on async lookup mode #20531
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the contribution, I left some comments
* | ||
* @return the UserDefinedFunction by preferable lookup mode, if require | ||
* <pre>{@code | ||
* 1. if upsertMaterialize == true : require sync lookup orElse error |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or else ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
this.userLookupFunction = userLookupFunction; | ||
this.retryStrategy = retryStrategy; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
please do some null check for given arguments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will add checkNotNull
public void open(FunctionContext context) throws Exception { | ||
super.open(context); | ||
userLookupFunction.open(context); | ||
retryResultPredicate = | ||
retryStrategy.getRetryPredicate().resultPredicate().orElse(ignore -> false); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: the open method can be moved to after constructor
}); | ||
|
||
@Test | ||
public void testLookupWithRetry() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could you add more tests to cover the branches:
- retryEnabled is disabled
- different ResultRetryStrategy
- ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will add tests cover the case retry disabled, and also a custom result retry strategy using exponential backoff delay (though we only support fixed delay in sql for now)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@godfreyhe thank you for reviewing this! I've addressed all your comments and updated this pr.
* | ||
* @return the UserDefinedFunction by preferable lookup mode, if require | ||
* <pre>{@code | ||
* 1. if upsertMaterialize == true : require sync lookup orElse error |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok
this.userLookupFunction = userLookupFunction; | ||
this.retryStrategy = retryStrategy; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will add checkNotNull
}); | ||
|
||
@Test | ||
public void testLookupWithRetry() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will add tests cover the case retry disabled, and also a custom result retry strategy using exponential backoff delay (though we only support fixed delay in sql for now)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…nc lookup mode This closes apache#20531
What is the purpose of the change
FLINK-28848 introduces a new LOOKUP hint to support retry both on sync and async lookup, but the async path doesn't works because of two reasons: 1. AsyncLookupJoinRunner has a 'stateful' resultFutureBuffer (which is very important for performance) bind to each input record (it's non-reenter-able) 2. can not lookup new value if cache empty values enabled when chained with the new AsyncCachingLookupFunction. So this can not be done by directly enabling retry on 'AsyncWaitOperator', so similar to sync lookup join with retry, we fix this by using a 'RetryableAsyncLookupFunctionDelegator' to support retry.
Brief change log
Verifying this change
Does this pull request potentially affect one of the following parts:
Documentation