-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nested mapTasks invocations can cause deadlock #4
Comments
This issue is split off from #2. |
Raising an exception when calling My expectation is that any finite poset of jobs should eventually complete. This implies that a if a worker runs a job that calls |
@depp, unfortunately that's not always correct, as the waited on
Because of this new distinction any solution would require some consideration. |
This would require us to somehow figure out whether we're run from one of the tasks, otherwise that would spawn one more thread than requested by caller. |
a dumb fix not accounting for whether we're run from one of the tasks that would spawn one more thread than requested by caller fixes jwiegley#4
a dumb fix not accounting for whether we're run from one of the tasks that would spawn one more thread than requested by caller fixes jwiegley#4
thanks, @l29ah |
Pinging @spl
If
mapTasks
invokesmapTasks
, deadlock can occur. Scenario:mapTasks
, which enqueues all of its jobs immediately in the execution graph, starts executing the first N tasks. It also waits until all tasks are finished before returning to the caller.mapTasks
, it will enqueue M more tasks, and wait for them to complete as well. However, the innermapTasks
can starve because no execution slots may ever free up: all jobs are now blocked, waiting on future jobs, which are waiting on the blocked jobs to finish.Solution: If we notice a call to
wait
in a thread whosethreadId
matches that of a running job, raise an exception that there may be a deadlock scenario.We could either raise this exception always (which makes it easier for the developer to avoid this situation), or we could only raise it if, when
wait
is called, there are no available slots.The text was updated successfully, but these errors were encountered: