Haproxy terminate socket connection when reloaded by Consul Template #525
Labels
Comments
|
Hi @brunomacf Consul Template is executing a subprocess, so I think this might be an haproxy issue itself. This also could be a duplicate of #442 manifesting itself differently. Unfortunately I am not familiar with thrift sockets with respect to haproxy. Are you able to reproduce this issue without Consul Template? That will determine if this is a CT issue or haproxy one. |
|
Hi @sethvargo As you pointed, it seems that the problem is more with haproxy and how it reloads (not so gracefully in my opinion) than with consul template. But one thing comes to my mind: my logs show that consul template are reloading haproxy very often, even if my containers stays the same. It would be great if consul templates reloads haproxy config file only when there is a change in consul context (e.g, some new container is registered or goes down). Is there some configuration of this kind yet? Thanks a lot to your reply. |
|
Hi @brunomacf Consul Template will only execute the command if the data has changed. What does your template look like? |
|
My haproxy template is exactly this: But what i'm seeing here is haproxy been reload very often (like 10 to 10 seconds). The logs are (using docker-compose to run everything in conjunction): PS. 1: Unfortunately my docker-compose do not log timestamps, but the "Retry connection" pops up from 10 to 10s as i said :) PS. 2: When i connect service2 to service1 directly (without haproxy) the reconnection tries go away as expected and therefore the problem must be with haproxy reloads. |
|
Hi @brunomacf Can you please show the Consul Template logs in debug mode? It will tell you why the template is being re-rendered and which dependencies are changing. Those logs don't indicate CT is restarting the processes to me, just that the process is being restarted. |
|
The logs since haproxy container initialization is shown bellow (consul-template in debug mode). The service account_v1 registered in consul (through gliderlabs registrator) is the "service2" that i was talking above and it is accessed by an rest api container. The "Consul returned 1 services" is quite often, right? Is this normal even if my services are not changing at all? I don't know whether this is breaking my thrift sockets connections or something else! PS. 15263: Sorry my bad english... it sucks! :) |
|
Hi @brunomacf That log output does not show any indication Consul Template is restarting the process. When Consul Template executes the command, it will have a log line like "executing command "haproxy reload ..."" which I am not seeing in that output. The template you posted above and he log output here also do not seem to match. The log output from haproxy also doesn't seem to indicate the haproxy process is restarting, but rather that it cannot connect to the given address and is retrying the connection. |
|
@sethvargo i'm closing this issue because the problem was between the screen of my laptop and the chair in which i was sitting on (if you understand me).... in my haproxy configuration i set a client timout of 10s and therefore haproxy was closing conmections after this time. Thanks very much for the support man! :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm using consul template to provide haproxy config file which load balance a set of thrift services. The configuration file for consul-template is:
and for haproxy is:
The problem is that every time consul template reloads the haproxy config file the thrift socket connection passing by haproxy server is killed and all services retries to connect. Is there a workaround for this?
PS. 1: Everything is running through docker containers, but i think this is not a problem. :)
PS. 2: I'm running NodeJS services (nodejs 5.3.0), consul-template 0.12, haproxy 1.6)
The text was updated successfully, but these errors were encountered: