MLCOMPUTE-949 | Pick spark ui port from a preferred port range #128
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
edingroot
force-pushed
the
u/chi/pick_from_port_range
branch
from
e78ff28
to
f975cc9
Compare
edingroot
commented
Sep 15, 2023
nemacysts
reviewed
Sep 18, 2023
service_configuration_lib/utils.py
Outdated
| @@ -28,3 +35,52 @@ def load_spark_srv_conf(preset_values=None) -> Tuple[Mapping, Mapping, Mapping, | |||
| except Exception as e: | |||
| log.warning(f'Failed to load {DEFAULT_SPARK_RUN_CONFIG}: {e}') | |||
| raise e | |||
|
|
|||
|
|
|||
| def ephemeral_port_reserve_range(preffered_port_start: int, preferred_port_end: int, ip='127.0.0.1') -> int: | |||
There was a problem hiding this comment.
it'd be nice to upstream this into the ephemeral_port_reserve repo, but we can wait to see how this works for y'all's usecase first if you'd like :)
There was a problem hiding this comment.
Sure! I added comments explaining the purpose of adding this function and linked to the upstream repository for reference. Let's leave it here for now and test it out first
88manpreet
approved these changes
Sep 20, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pick the Spark UI (API) port from a preferred port range, for scraping metrics from multiple Spark sessions in a single Jupyter server.
Tested in a jupyter server and launched 11 Spark sessions, the port of the 11th session is a random ephemeral port that is currently available on the host/pod.