-
Notifications
You must be signed in to change notification settings - Fork 13.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-8887][flip-6] ClusterClient.getJobStatus can throw FencingTokenException #5648
Conversation
} else { | ||
return FutureUtils.completedExceptionally(new FlinkJobNotFoundException(jobId)); | ||
try { | ||
return jobManagerRunner.getJobManagerGateway().requestJobStatus(timeout); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is an asynchronous call that isn't throwing the exception. You have to add a handler to the returned CompletableFuture
. It also only properly resolves one of the exceptions, and IMO shouldn't catch Exception
but the specific exceptions we want the workaround to work for to not hide other issues.
In any case, I'm not sure if adding workarounds to the Dispatcher is the right way to go. These issues revealed that some scenarios are not properly handled, and I would prefer waiting for @tillrohrmann to really fix this in the Dispatcher and related components.
We can temporarily handle both exceptions in the MiniClusterClient
by adding a single retry (with a short sleep) if a specific exception occurs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, I will try this way, thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think methods such as requestJob
and requestOperatorBackPressureStats
are equally affected by this bug. I would also like to hear @tillrohrmann's opinion first before proceeding.
hi @zentol how about this solution? I think try another way(fetch from archived execution graph store) from CompletableFuture.handler is a good choice. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are now somewhat doing correctly what you originally intended, but it still doesn't solve all problems as I said in my first comment.
The JobManager can still receive messages before the fencingToken was set (which causes the first exception in the JIRA). There's also another message that can sometimes occur where the JM RpcEndpoint wasn't started yet. Both of these can be fixed with a retry, as is the issue that you're currently addressing.
if (throwable != null) { | ||
Throwable error = ExceptionUtils.stripCompletionException(throwable); | ||
|
||
if (error instanceof FencingTokenException) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
missing else block causes other exceptions to be swallowed...
Already fixed. |
What is the purpose of the change
This pull request fixed ClusterClient.getJobStatus throw FencingTokenException issue
Brief change log
Verifying this change
This change is already covered by existing tests, such as DispatcherTest.testCacheJobExecutionResult.
Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes / no)Documentation