Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace register_worker_callbacks with worker plugins #2453

Merged
merged 10 commits into from May 22, 2019

Conversation

@mrocklin
Copy link
Member

commented Jan 8, 2019

These provide space for a bit more complexity, including teardown and optional idempotence. This is still a work in progress. Some things I'd like to do:

  • deprecate register_worker_callbacks
  • support simple setup functions in register_worker_plugins
  • improve error handling in broadcast

cc @rainwoodman and @guillaumeeb . Possible replacement for #2391

distributed/client.py Outdated Show resolved Hide resolved
@rainwoodman

This comment has been minimized.

Copy link
Contributor

commented Jan 8, 2019

I like this better than #2391.

@guillaumeeb
Copy link
Member

left a comment

I agree with @rainwoodman, this design looks better!

distributed/client.py Outdated Show resolved Hide resolved
distributed/client.py Outdated Show resolved Hide resolved
distributed/client.py Outdated Show resolved Hide resolved
distributed/client.py Outdated Show resolved Hide resolved
distributed/worker.py Show resolved Hide resolved

mrocklin added some commits May 21, 2019

@mrocklin

This comment has been minimized.

Copy link
Member Author

commented May 21, 2019

@jcrist should you find yourself with a moment this could also use your review

mrocklin added some commits May 22, 2019

@mrocklin

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

Docs added in dask/dask#4833

@mrocklin

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

I plan to merge this this afternoon if there are no further comments

def register_worker_plugin(self, plugin=None, name=None):
"""
Registers a lifecycle worker plugin for all current and future workers.

This comment has been minimized.

Copy link
@TomAugspurger

TomAugspurger May 22, 2019

Member

General question: do you have thoughts on using ..versionadded:: <version> directives? Don't need to do it here necessarily, but I ask since this is being referenced from the dask documentation, which may be on a different release cycle.

This comment has been minimized.

Copy link
@mrocklin

mrocklin May 22, 2019

Author Member

No strong thoughts. I've never used them much, either as an author or reader.

My plan was just to merge the doc PR after this gets released.

Happy with whatever though.

This comment has been minimized.

Copy link
@mrocklin

mrocklin May 22, 2019

Author Member

Ah, I see that you've since merged the doc PR :) Shouldn't be a big deal either way

@TomAugspurger
Copy link
Member

left a comment

Small question on a test name. Since Github / travis have been having issues with web hooks today, you may just want to merge.



@gen_cluster(client=True, worker_kwargs={"plugins": [MyPlugin(5)]})
def test_idempotence_with_name(c, s, a, b):

This comment has been minimized.

Copy link
@TomAugspurger

TomAugspurger May 22, 2019

Member

Are the names of these two tests flipped (this one doesn't use plugin names, the other does?)

This comment has been minimized.

Copy link
@mrocklin

mrocklin May 22, 2019

Author Member

I think that we're good here. The MyPlugin class has a name attribute, which is what causes the idempotence here.

In the test below, we change the name intentionally.

This comment has been minimized.

Copy link
@TomAugspurger

TomAugspurger May 22, 2019

Member

Ah, understood.

@TomAugspurger TomAugspurger merged commit 6134c75 into dask:master May 22, 2019

1 of 2 checks passed

continuous-integration/appveyor/pr AppVeyor build failed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@mrocklin mrocklin deleted the mrocklin:worker-plugin branch May 22, 2019

@mrocklin

This comment has been minimized.

Copy link
Member Author

commented May 22, 2019

Woot. Thanks for merging @TomAugspurger ! I think that some version of this PR has existed for years now :)

calebho added a commit to calebho/distributed that referenced this pull request May 29, 2019

Replace register_worker_callbacks with worker plugins (dask#2453)
* Add worker plugins

* add docstring

* Replace legacy worker_callbacks with worker_plugins

* add and test name keyword

* fix missing import

* black

* respond to feedback

* Handle errors again

* Expand docstring
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.