-
Notifications
You must be signed in to change notification settings - Fork 61
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
race condition between consul-template
and restarting haproxy
#69
Comments
OK, browsing the internet a little bit more and reading stuff, I came to the conclusion, that multiple running But then there might be another issue for this observed behavior: As soon as there is a change with respect to services running and a new Any ideas? |
ad. to HAproxy multiple processes - this is a normal behavior. Do you have problem with every running service, when HAproxy reloads ? |
I just setup a very simple This service gets stressed by another As soon as the
or sometimes connection timeout like this:
|
I think this is kind of HAproxy design problem that for few millisecond port is unavailable - specially exposed if you test it without keep alive - you should rather simulate browser behavior and check We are considering to migrate to different Load Balancer. One of our internal developer wrote a golang consul based router, we are waiting for OS it (It need some time to Open Source any internal tool) |
if this is still bothering you, change in
I just saw that this option cannot be overwritten from external ENV variable , man man man |
Thx for the pointers; will have a closer look tomorrow in the office. I hoped/thought that HAProxy could handle those cases out-of-the-box... :-( Do you have experience with nginx? Can it handle that reload case more gracefully? |
Yes we have experience with nginx, apache and varnish. IMHO all those methods doesn't fully feet to consul cluster - because you always need to use consul-template "glue code" thats why some of my colleagues decided to write its own router. |
I have just tested solution with |
Yes, definitely! :-) Thank you for all your suggestions and hard work! |
fixed |
thx |
This change didn't fix the issue for me; I am observing the exact same errors as noted above... :-( |
Yea, I forgot to include one part, I was wondering where to put it, and forgot about it. |
oh man this is not working as suspected, depends on buffer size it can even slow down a lot. |
I have a PoC using iptables switching between two HAproxy, |
Kindly please verify if this is working fine |
This seems to work now and also with my Thanks for that. |
It looks like there is a race condition between
consul-template
andrestarting haproxy
. I noticed on several cluster nodes thathaproxy
is running several times, e.g.:Note, that there are more than one
pid
s listed for the-sf
option ofhaproxy
!My guess is, that this happens, when there are several service changes propagating through
consul-template
with very short time intervals in between, e.g., when scaling amarathon
app. This will trigger the haproxy reload command in parallel each one getting one or morepid
s via thepidof
call resulting in the situation given above.The result of this is that
haproxy
on that mode will/might run with an outdatedhaproxy config
.Right now, I don't have a good idea how to fix that...
The text was updated successfully, but these errors were encountered: