This repository has been archived by the owner on May 15, 2020. It is now read-only.
Monitor SageMaker updates #113
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ksaaskil
force-pushed
the
monitor-sagemaker-updates
branch
from
17a05c4
to
e4049e8
Compare
This reverts commit 1ee6f59.
idantene
reviewed
Jan 14, 2019
| @@ -197,6 +197,7 @@ def get_updates(self, job_id: uuid.UUID) -> HistoryByScalar: | |||
| @Pyro4.expose | |||
| def stop(self): | |||
| if self.__was_shutdown: | |||
| LOGGER.warning("Agent API was shutdown twice.") | |||
| @@ -91,8 +106,8 @@ def get_job_status(self, job_name) -> JobStatus: | |||
| :raises JobNotFoundException: If job was not found. | |||
| :return: Job status | |||
| """ | |||
|
|
|||
| self.check_or_build_connection() | |||
| with self.lock: | |||
There was a problem hiding this comment.
Why a thread lock? Are there any race conditions?
| LOGGER.exception("Polling for updates failed") | ||
| # Ignore | ||
|
|
||
| # TODO Remove duplicate code with Scheduler |
There was a problem hiding this comment.
Goes back to perhaps having a common base class with Scheduler 🤔
| @@ -66,6 +67,8 @@ def daemonize(self, serialized): | |||
| pid = os.fork() | |||
| if pid > 0: # Close parent process | |||
| return | |||
| if not self.terminate_daemon: | |||
| self.terminate_daemon = multiprocessing.get_context("spawn").Event() | |||
There was a problem hiding this comment.
multiprocessing.get_context("spawn") should also be abstracted away somehow so we don't duplicate it everywhere
Merged
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sorry for the huge PR, it took quite some effort to get stuff working.
sagemaker.analyticsappends new rows to the end of DataFrame once new values emerge. This should be checkedjob.add_scalarSchedulerfor querying and posting job metrics, these should probably go toTrackerBaseorJobMonitorBaseto avoid duplicating codethreading.Lockin initializingSageMakerHelperconnections to avoid race conditionsSageMakerJobMonitor. It only notifies when new values are available, though.meeshkan stopto not raise an exception when used before starting the agentterminate_daemonevent in the daemon process, this should be more robust than setting it before spawning and forking (?)maxParallelto one. Also changedterminate_daemonto be created in the API process. The process can only be stopped via the Pyro proxy anyway in real usage, as only tests re-use theServiceinstance forstart->stopetc.