Replace register_worker_callbacks with worker plugins #2453
Conversation
|
I like this better than #2391. |
|
@jcrist should you find yourself with a moment this could also use your review |
|
Docs added in dask/dask#4833 |
|
I plan to merge this this afternoon if there are no further comments |
| def register_worker_plugin(self, plugin=None, name=None): | ||
| """ | ||
| Registers a lifecycle worker plugin for all current and future workers. | ||
|
|
There was a problem hiding this comment.
General question: do you have thoughts on using ..versionadded:: <version> directives? Don't need to do it here necessarily, but I ask since this is being referenced from the dask documentation, which may be on a different release cycle.
There was a problem hiding this comment.
No strong thoughts. I've never used them much, either as an author or reader.
My plan was just to merge the doc PR after this gets released.
Happy with whatever though.
There was a problem hiding this comment.
Ah, I see that you've since merged the doc PR :) Shouldn't be a big deal either way
|
|
||
|
|
||
| @gen_cluster(client=True, worker_kwargs={"plugins": [MyPlugin(5)]}) | ||
| def test_idempotence_with_name(c, s, a, b): |
There was a problem hiding this comment.
Are the names of these two tests flipped (this one doesn't use plugin names, the other does?)
There was a problem hiding this comment.
I think that we're good here. The MyPlugin class has a name attribute, which is what causes the idempotence here.
In the test below, we change the name intentionally.
|
Woot. Thanks for merging @TomAugspurger ! I think that some version of this PR has existed for years now :) |
* Add worker plugins * add docstring * Replace legacy worker_callbacks with worker_plugins * add and test name keyword * fix missing import * black * respond to feedback * Handle errors again * Expand docstring
These provide space for a bit more complexity, including teardown and optional idempotence. This is still a work in progress. Some things I'd like to do:
register_worker_callbacksregister_worker_pluginscc @rainwoodman and @guillaumeeb . Possible replacement for #2391