You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When a worker offers multiple resources, e.g. GPU=2 CPU=4, a task may remain stuck in constrained state despite its resource being available.
The use case is:
a task requiring a depleted resource is enqueued on the worker
a task requiring a non-depleted resource is enqueued afterwards. The task should transition to executing immediately; instead it remains in the queue.
This is because Worker.constrained is a single deque for all tasks with resources, which will stop at the first task whose resources can't be satisfied.
Reproducer:
@gen_cluster(client=True, nthreads=[("", 2, {"resources": {"A": 1, "B": 1}})])asyncdeftest_multi_resources(c, s, a):
# Let all tasks pile up in the Worker before they can be processedevc=Event()
clog=c.submit(lambdaev: ev.wait(), evc, resources={"A": 1, "B": 1}, key="clog")
evx=Event()
x1=c.submit(lambdaev: ev.wait(), evx, resources={"A": 1}, key="x1")
x2=c.submit(lambdaev: ev.wait(), evx, resources={"A": 1}, key="x2")
y=c.submit(inc, 1, evx, resources={"B": 1}, key="y")
whilelen(a.tasks) <4:
awaitasyncio.sleep(0.01)
asserta.tasks["clog"].state=="executing"asserta.tasks["x1"].state=="constrained"asserta.tasks["x2"].state=="constrained"asserta.tasks["y"].state=="constrained"awaitevc.set()
# x1 constrained -> executing# x2 remains in queue as it uses the same resource as x1# y constrained -> executing, despite being lower priority than x2, because it uses# a different resourceawaityawaitevx.set()
awaitwait([x1, x2])
When a worker offers multiple resources, e.g.
GPU=2 CPU=4
, a task may remain stuck in constrained state despite its resource being available.The use case is:
This is because Worker.constrained is a single deque for all tasks with resources, which will stop at the first task whose resources can't be satisfied.
Reproducer:
The above reproducer hangs on
await y
.Related issues:
The text was updated successfully, but these errors were encountered: