Add PipInstall WorkerPlugin

[mrocklin]

Examples

>>> from dask.distributed import PipInstall
>>> plugin = PipInstall(packages=["dask", "scikit-learn"])
>>> client.register_worker_plugin(plugin)

[mrocklin]

cc @TomAugspurger @rabernat @jhamman

[TomAugspurger]

Possibly would want to accept an argument for options passed to pip. Things like --upgrade, --pre, etc.

So perhaps a pip_options=None that's a list of strings, just like in Popen.

[TomAugspurger]

Pushed a few changes here.

@jcrist do you have any objections to the overall idea of doing this in a worker plugin?

[mrocklin]

Conceptually there are issues here around restarting. We want to restart things if we have different versions than what we just installed. However it's also possible that another process just updated our versions, and we should restart, even though when we call `pip install` everything seems like it is already installed.

…

On Tue, Nov 19, 2019 at 2:25 PM Tom Augspurger ***@***.***> wrote: Pushed a few changes here. @jcrist <https://github.com/jcrist> do you have any objections to the overall idea of doing this in a worker plugin? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3216?email_source=notifications&email_token=AACKZTCLRZF7PR24CWYRQEDQURKTBA5CNFSM4JLJRQM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEP2MIA#issuecomment-555722272>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTBX47EL4MTD4XPRRX3QURKTBANCNFSM4JLJRQMQ> .

[kyprifog]

Whats the status of this effort? I just ran into an issue running dask-gateway with not having s3fs on the workers and I'm trying to find the path of least resistence.

[TomAugspurger]

I pushed a couple updates to

• fix merge conflicts
• add a test for an install failures (e.g. package typo)
• added the PipInstall plugin to the docs

I think this should be good to go.

[mrocklin]

I think that there are likely still issues when you have multiple workers on the same machine pip/conda installing things. However, this may be an uncommon enough case where this is likely to be used that we shouldn't care. I'm not sure. I think that it's totally ok to merge anyway.

[kyprifog]

@mrocklin, to clarify are you saying there are issues with multiple clients trying to issue (possibly contradictory or overlapping) commands to the workers of the same cluster to update packages?

@TomAugspurger I am trying to test this (rudimentarily) by doing this. Am I doing it wrong? I am still getting a package not found s3fs error from the workers:

with Client() as client:
    plugin = PipInstall(packages=["s3fs"], pip_options=["--upgrade"], restart=True)
    client.register_worker_plugin(plugin)
    df = dd.read_csv(files)
    df.count().compute()

>>
dask-worker-4c0dd1f1d56b4050ba4fc4e1e89edd44-vb7cg dask-worker ModuleNotFoundError: No module named 's3fs'

[mrocklin]

I'm saying that if multiple workers are on the same filesystem (such as happens when you say dask-worker --nprocs 2) then they will both try to pip/conda install and both try to restart, sort of. Things can fail easily in that situation. What you're suggesting is also an issue, but one that seems much less likely to occur.

…

On Thu, Oct 1, 2020 at 11:58 AM Kyle Prifogle ***@***.***> wrote: @mrocklin <https://github.com/mrocklin>, to clarify are you saying there are issues with multiple clients trying to issue (possibly contradictory or overlapping) commands to the workers of the same cluster to update packages? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3216 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTBKBKG2LUJVMOSKF33SITGN7ANCNFSM4JLJRQMQ> .

[TomAugspurger]

I am still getting a package not found s3fs error from the workers:

I wonder if the installation / restart fails to complete before you start your computation requiring s3fs?

The issue with multiple workers sharing a filesystem is a good one I hadn't considered. I'm OK with just documenting that shortcoming for now.

[kyprifog]

I am still getting a package not found s3fs error from the workers:

I wonder if the installation / restart fails to complete before you start your computation requiring s3fs?

The issue with multiple workers sharing a filesystem is a good one I hadn't considered. I'm OK with just documenting that shortcoming for now.

I had wondered the same thing but if that were the case running the same operation twice would result in a positive response. That being said, it didn't appear that the workers restarted in kubernetes so maybe I am doing something else wrong. I think maybe there is also a chance something weird with conda is happening, the command that finally worked when run on each worker was:

conda install --yes -c conda-forge s3fs==0.4.2

So i had to use conda and downgrade s3fs (I was getting an error form fsspec about "other_paths" with 0.5.0 and 0.5.1)

For context, I'm trying to avoid forking the helm chart for daskhub which is why I'm spending more time trying to figure out how to do this via plugin.

Its possible the issue is actually related to conda environments.

[kyprifog]

Update: I was able to get my issue resolved by building my own conda install worker plugin.

[TomAugspurger]

That being said, it didn't appear that the workers restarted in kubernetes

What do you mean by this? I don't believe this would be visible at the kubernetes level (other than via logs), since no pods / processes are going away.

[kyprifog]

Yeah I meant via logs, I was running stern dask and non of the workers logged any changes. On my own plugin, I just ended up doing it synchronously and not restarting the workers bc I couldn't figure out what was going on and it seemed to work. 🤷 Maybe it was actually working, dask-gateway was just expecting conda install, idk.

[TomAugspurger]

Gotcha, thanks for clarifying.

[TomAugspurger]

I'm planning to merge this later today.

[mrocklin]

+1

…

On Fri, Oct 9, 2020 at 9:07 AM Tom Augspurger ***@***.***> wrote: I'm planning to merge this later today. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3216 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTBONRJMLMVTLPQUZH3SJ4YLJANCNFSM4JLJRQMQ> .