Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Duplicate task execution when running long tasks in parallel #487
I often get duplicate task execution problem when running many tasks in parallel with following conditions.
Steps to reproduce
In this case, task
I think the cause of this problem seems to be how to start the task execution thread on MultiThreadAgent#run.
ThreadPoolExecutor#getActiveCount returns approximate active number of threads.
To testing this problem, I add some info logs to MultiThreadAgent#run as follows.
The execution result is:
And at this time,
In ths case, task with id 160 (task
And I have confirmed that updating
This may cause the duplicate task execution.
digdag version: 0.9.4
MultiThreadAgent is the only instance that can increase number of active threads. There're no other threads that may start another thread right after
So, solution would be something like this: instead of checking number of active threads, MultiThreadAgent needs to monitor number of active (non-finished) tasks.