Poll prior to RUNNING state #71
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In order to look into improving kernel startup times, we should attempt to get connection information from the assigned host as soon as the host is known. This is because connection information could be made available even before Yarn has placed the application into RUNNING state since spark-submit jobs require a spark session for that state transition to occur. By attempting to get that information, we have the ability to look at possible asynchronous session creation, etc. I also took this opportunity to refactor the pull and socket code into local methods so as to clean up the application startup logic. This included reformatting debug messages such that useful information is more quickly visible.
Decreased the non-interruptable sleep timeout from 1 to .5 seconds so that we check status of connection information more frequently. Increased interruptable socket timeout from 1 to 5 seconds so that we optimize socket mode operations since the method will be interrupted when the data arrives. This way we don't spend time waiting on non-interrupted calls. Also performed a little more refactoring to cut down on the number of Yarn queries performed each iteration.
This was referenced Jul 13, 2017
|
It's more clean after handling pull/socket mode are refactored from the LGTM. Merge it now. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These changes address Issue #69 and are related to Issue #64.
To optimize the time-to-connect duration following a yarn-cluster kernel's launch, we now start polling for connection information as soon as host assignment occurs. This way launchers can create the connection information prior to creating the spark session - which allows the kernel to reach a connected state as soon as possible.
Other changes: