[FLINK-9211][REST] JarRunHandler submits job to Dispatcher via RPC #5903
Conversation
| }); | ||
|
|
||
| CompletableFuture<Acknowledge> jobSubmissionFuture = jarUploadFuture.thenCompose(jobGraph -> { | ||
| // we have to enable queued scheduling because slots will be allocated lazily |
There was a problem hiding this comment.
Is this also what the thing sitting behind the REST endpoint does?
There was a problem hiding this comment.
enabling queued scheduling
There was a problem hiding this comment.
If by the thing sitting behind the REST endpoint you mean the Dispatcher, then no, it does not enable queued scheduling, but effectively relies on it being enabled. If you don't enable it beforehand in the client/handler the job submission will outright fail with an entirely unhelpful error message.
The Dispatcher should probably enable it though, but that isn't really in the scope of this PR as it would also affect the RestClusterClient which does the same thing.
Technically it may also be questionably as it would invalidate part of the JobGraph API.
There was a problem hiding this comment.
gotcha. I think that's way off topic, so this is still good to merge
|
merging. |
What is the purpose of the change
This PR reworks the
JarRunHandlerto submit the job to the dispatcher via RPC, instead of setting up aRestClusterClientand going through the client's job-submission routine.The reasoning is that the existing behavior was causing issues on kubernetes, and this change also removes a special-case as this was the only handler that actively sends out rest requests.
Brief change log
JarRunHandlernow has access toDispatcherGatewayJarRunHandlernow uploads jar and submits job via RPCVerifying this change
JarRunHandlerTestDoes this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation