Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Parallel execution is slower than sequential exectuion for trivial tasks. #868
First of all I would like to thank you for developing this great tool, it really makes my life much easier.
I encounter a strange problem when using fabric in parallel mode: for trivial jobs like getting hostname, sequential execution on multiple hosts are much faster than parallel mode. I didn't find anything in the documentation to configure to speed up the process and I didn't find anyone having similar issues on the Internet. After reading the code, I found that this is caused by the JobQueue.__fill_results() trying to retrieve additional results from the _comms_queue. It is called both in the main loop processing jobs as well as after the main loop. The functions use a timeout of 1 seconds as in version 1.6 which causes unnecessary long delay when executing trivial jobs.
I am wondering whether there is any specific reason to use 1 second timeout. For my application, I need it to be fast so I plan to change it to 0.01 second timeout for the statement
Will there be any issues? Thank you very much!
If you can look at the Git history for those lines it might be useful re: finding out why 1s was used as the default. Many other situations have an 0.1 or 0.01 or etc, loop sleep time -- see e.g.