Add --env to dask-worker, support lists for --nprocs

[mrocklin]

We might consider adding an --env keyword to dask-worker that passed values down to the env= parameter of Nanny. (in the no-nanny case, this might fail).

Additionally in the case that this is used with --nprocs, (this is sort of hacky) it would be useful if this parameter would also support a list of inputs, so that people could pass in different environment variables for each of their processes.

This came up in rapidsai/dask-cuda#3 (comment)

I'm not sure how I feel about it.

[quasiben]

env would also be helpful in the context of UCX where we are often setting UCX_MEMTYPE_CACHE=n. I'd be happy to take this on next week

[mrocklin]

Are you using the dask-worker CLI command, if so, why not use export ... just before?

[quasiben]

When using the dask-worker CLI I do use export or prefix env var prior to dask-worker ... but in the case of using LocalCluster passing env vars around might be helpful

[mrocklin]

You should already be about to use an env= dict though. Extra key words to LocalCluster pass through to Nanny.

…

On Sun, Apr 14, 2019, 1:19 PM Benjamin Zaitlen ***@***.***> wrote: When using the dask-worker CLI I do use export or prefix env var prior to dask-worker ... but in the case of using LocalCluster passing env vars around might be helpful — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszIRGahQhYBza6ZxBPHjevqVSYPZTks5vg3FCgaJpZM4akTcX> .

[quasiben]

Hmm, perhaps I'm reading it incorrectly but I don't think we can pass env without errors. With something like the following

env={'UCX_MEMTYPE_CACHE': 'n'}
with LocalCluster(..., processes=False, env=env) as cluster:
   ...

I think we'll error out when we hit the Worker constructor here:
https://github.com/dask/distributed/blob/master/distributed/deploy/local.py#L287-L293 there will be an issue

[mrocklin]

This should have been added in #2431

…

On Mon, Apr 15, 2019 at 9:46 PM Benjamin Zaitlen ***@***.***> wrote: Hmm, perhaps I'm reading it incorrectly but I don't think we can pass env without errors. With something like the following env={'UCX_MEMTYPE_CACHE': 'n'}with LocalCluster(..., processes=False, env=env) as cluster: ... I think we'll error out when we hit the Worker constructor here: https://github.com/dask/distributed/blob/master/distributed/deploy/local.py#L287-L293 there will be an issue — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2508 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszFgd6fS0Q_algw70OZUXD_n5TNpjks5vhTlygaJpZM4akTcX> .