Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Minion continues trying down master #50814
Description of Issue/Question
In a scenario where you have a master of masters with syndic masters running in Hot,Hot it is possible to see a long delay in events from a minion after a master has gone down. The minion appears to continue to attempt to send events serially instead of marking the master as down and moving it to the end of the list.
Use the following state on your masters to be called on the minion to fire an expected event.
Overview of architecture
Master of Masters, 2 masters with the same syndic_master, 1 minion connected to both masters.
Steps to Reproduce Issue
As a result of the above, new jobs to this minion are delayed heavily, waiting for this to timeout.
I believe this issue is due to the code here
The code is still almost exactly the same in 2018.3.3, so I expect this problem to persist.
The problem appears to occur because