Sporadic Hang With TaskCollaboration Plugs With Many Threads #5403
Comments
|
Here is the contents of my test file, I've been running it like this: repeat 40 timeout -s KILL 30s gaffer stats deleteFacesHang3.gfr -nodeSummary 0 -threads 50 -scene Group.out This should produce a usual runtime of about 1 second scene generation ( plus several seconds loading ), but in the run of 40, you should see some take over 30 seconds and get killed by the timeout command. The stack trace shows that one thread gets into the parallel for within ShadingEngine::shade, which should only be running OSL stuff, but then somehow that hits a mutex and deadlocks ... our understanding is that this is because another thread has tried to help with the parallel_for, but is over max_concurrency, so has instead enqueued a task which gets picked up while running the parallel_for. |
|
Good hunting Daniel! I think there a couple of relatively easy ways to resolve this :
I'm fairly inclined to go with option 1 for its simplicity, and because I can't think of a good reason to oversubscribe. |
johnhaddon
added a commit
to johnhaddon/gaffer
that referenced
this issue
Aug 9, 2023
This fixes deadlocks caused by worker threads not being able to join a `tbb::task_arena`, because by default they are constructed to only allow the same number of workers as cores. Fixes GafferHQ#5403. Also added support for specifying negative thread counts, to reserve some cores for other applications.
johnhaddon
added a commit
to johnhaddon/gaffer
that referenced
this issue
Aug 11, 2023
This fixes deadlocks caused by worker threads not being able to join a `tbb::task_arena`, because by default they are constructed to only allow the same number of workers as cores. Fixes GafferHQ#5403. Also added support for specifying negative thread counts, to reserve some cores for other applications.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Summarizing this investigation which turns out to not be very high priority because it only occurs if the "-threads" argument is used to run Gaffer with more threads than cores.
The theory is that TaskMutex uses mechanisms like
executionState->arena.execute( [&executionState]{ executionState->taskGroup.wait(); } );for threads to join an arena - we expect this to join the arena, and immediately execute thetaskGroup.wait. However, if the arena is already considered "full", then this instead enqueues the "taskGroup.wait", which means that threads already in the arena could pick up this wait, causing a deadlock.I will include below a simple repro.
A guard that I tried out which did not prevent the deadlock, but did strongly indicate that something weird is happening with high thread counts, is putting this in TaskMutex::acquireOr:
This is triggered when running Gaffer with -threads set higher than the core count, indicating that there is the possibility of this wait running at a time we don't expect ( though this guard by itself does not prevent hangs, indicating that something else weird may be going on at high thread counts ).
John spotted this weird work-around indicates TBB folks being aware of the issue internally:
https://github.com/oneapi-src/oneTBB/blob/v2020.3/src/tbb/arena.cpp#L1031-L1048
( If they added the try_execute method mentioned in that comment, that would probably solve the issue for us ).
We haven't figured out why that workaround appears to no longer be needed in bleeding edge TBB, nor can we find the forum post that John mentions in TaskMutex.h as being the original inspiration for this approach.
As far as we can tell, this issue currently only occurs when using -threads to run with more threads than cores ( if we don't over-subscribe at all, then we never run out of max_concurrency in the arena, and never trigger this behaviour from tbb ), so this may not be too high priority.
Maybe tbb will eventually add try_execute, or maybe on new tbb we'll be able to just replace TaskMutex with collaborative_call_once?
The text was updated successfully, but these errors were encountered: