Orchestrator running in HAProxy has intermittent runtime errors

[jpswinski]

[info] 318/133201 (1) : Loading orchestrator...
[NOTICE] (1) : New worker (8) forked
[NOTICE] (1) : Loading success.
Lua applet http '<lua.orchestrator_lock>': [state-id 0] runtime error: execution timeout from /usr/local/etc/haproxy/orchestrator.lua:256: in function line 226.
[ALERT] (8) : Lua applet http '<lua.orchestrator_lock>': [state-id 0] runtime error: execution timeout from /usr/local/etc/haproxy/orchestrator.lua:256: in function line 226.
Lua applet http '<lua.orchestrator_lock>': [state-id 0] runtime error: execution timeout from /usr/local/etc/haproxy/orchestrator.lua:249: in function line 226.
[ALERT] (8) : Lua applet http '<lua.orchestrator_lock>': [state-id 0] runtime error: execution timeout from /usr/local/etc/haproxy/orchestrator.lua:249: in function line 226.

[jpswinski]

This was a bug in the orchestrator code. When the number of locks was max'ed out, the code looped back to the beginning of the list of nodes yet set the index incorrectly and entered an infinite loop. HAProxy then killed it because it ran longer than the timeout.

The result of the bug was that locks were not unlocked and would have to timeout. This also slowed things down a lot as there weren't as many available locks.