Clarification on worker resources documentation

[chrisroat]

In the worker resources docs, it shows the following scenario.

dask-worker scheduler:8786 --nprocs 3 --nthreads 2 --resources "process=1"

With the code below, there will be at most 3 tasks running concurrently and each task will run in a separate process:

from distributed import Client
client = Client('scheduler:8786')

futures = [client.submit(non_thread_safe_function, arg,
                         resources={'process': 1}) for arg in args]

If additional tasks are scheduled without the process resource, might they be scheduled on the unused threads? The scenario I'm concerned about is when the {process: 1} consumes multiple threads, effectively starving these additional tasks.

If this is so, this leads me down the road of tagging all my additional tasks (and other workers) to avoid this situation. Consequently, I then lose out on scheduling on the process worker with these tasks when no process tasks are running. (Is this clear?)

I'd be happy to update the documentation to help elucidate this situation, once I feel I fully grok what is going on.

[TomAugspurger]

If additional tasks are scheduled without the process resource, might they be scheduled on the unused threads?

I think you're correct, but it'd be good to verify with tests / going through the code.

[chrisroat]

Anecdotally, I was finding this to be true. The trick of defining resources for all workers and all tasks can mitigate the problem.

However, this leads to under-utilization because an unused thread-hostile worker will not take up thread-safe work. Additionally, heavy use of resources requires turning off graph optimization for scheduling when intermediate steps need resources.

I added a test+doc to hopefully clarify the situation. Please let me know if anything is unclear.

Would it be a plausible to consider special task-only resources that could take-over an entire or partial worker? I am thinking along the lines of special resources named {"PROCESS": 1} or {"THREADS": 4}.

[mrocklin]

Would it be a plausible to consider special task-only resources that could take-over an entire or partial worker? I am thinking along the lines of special resources named {"PROCESS": 1} or {"THREADS": 4}.

Yes, it would be plausible to consider that. That being said, I don't immediately have a plan of how it would be implemented. Someone would have to put some thought into it, propose a plan, and then implement it.