[stable/dask] ability to set worker --nthreads and --nprocs independently of cpu resource limit

[skeller88]

A dask worker node should be able to have --nthreads and --nprocs set via a user-provided helm chart, and have those values be independent of worker cpu resources. One use case is high I/O workloads that benefit from more threads than cores.

Currently --nthreads is set to the worker cpu resource limit.

Aside from limiting dask helm deployment options, this limitation is problematic because it leads users to believe that there needs to be a 1-1 relationship between cpus and threads, which is not true.

A solution is to have a new option in the helm chart:

worker:
    args:
        nthreads: 6
        nprocs: 2

Other worker command line options could be added in this way.

[leoxtc]

So currently I can not explicit set the nthreads from the helm chart config ?

[skeller88]

Correct. You can use the feature branch of my fork of this repo in the meantime: https://github.com/helm/charts/compare/master...skeller88:add-dask-num-worker-threads?expand=1 Clone my fork of chart repo to [chart-repo-dir] and then to deploy: helm install -f helm_dask_chart.yaml ~/[chart-repo-dir]/stable/dask

…

On Tue, Nov 19, 2019 at 11:56 AM leoxtc ***@***.***> wrote: So currently I can not explicit set the nthreads from the helm chart config ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18708?email_source=notifications&email_token=ABKAKATGFZEFV5H4ONYQIGTQURAIBA5CNFSM4JKSZXY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEPQ33A#issuecomment-555683308>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKAKARNGGUOETBMKMW2QGLQURAIBANCNFSM4JKSZXYQ> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.