Better Dask integration #23
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
+ Added Delegate#job_complete function to support new methods of task tracking in the DaskDelegate. ~ Changed job creation to fire-and-forget. This ensures that Dask will not depend on the actual status of the Future to run. ~ Changed results retrieval to be entirely redis-dependent. ~ Updated DaskDelegate#job_status to use TaskState#state values. - Disabled LocalCluster support due to changes in how delegates are instantiated in the API. - Removed dependence on local future maps -- this no longer works due to API changes.
mdip226
reviewed
Jul 7, 2021
stochss_compute/api/api.py
Outdated
| :param port: The port to listen on. | ||
| :type port: int | ||
|
|
||
| :param delegate: The type of delegate to be used to run simulation jobs. Defaults |
There was a problem hiding this comment.
:param delegate_type:
:type delegate_type: type
or something like that, maybe?
There was a problem hiding this comment.
Thanks for catching that -- fixed!
mdip226
approved these changes
Jul 7, 2021
~ Changed start_api() docstring to reference `delegate_type`, not `delegate`.
mdip226
approved these changes
Jul 7, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains numerous changes that (hopefully) improve the behavior of Client <--> Compute <--> Redis interactions.
Changes
Delegateinstances are no longer constructed once on API start. Instead, they're dynamically created and destroyed based on Flask request sessions.DaskDelegateto no longer depend on internal state to track running jobs. Instead, the Dask Scheduler is queried for the state and, if completed, the results are grabbed from Redis.Futureinstances are no longer needed to be kept around to ensure that their corresponding job is run on a Worker.Delegate#job_completefunction to support new workflow.__init__.pyand intoapi.py.Closes #18