True Zero Downtime HAProxy Reloads

[modius]

Any chance we can get HAProxy reloads that do not drop traffic?

When services or service backends are added or permanently removed, HAProxy has to reload across our entire infrastructure. These reloads can cause reliability problems because while HAProxy is top notch at not dropping traffic while it is running, it can (and does) drop traffic during reloads.
http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html

[amir20]

I can confirm that connections are dropped while haproxy is reloading config.

I created a new service with haproxy. I set the API to full access and confirmed that when I switch between two links it does switch over correctly. When I ran siege over some time I noticed that about 4 connections were dropped. In a one minute test, that put it to around 99.86% uptime.

I can share my scripts if others want to see it.

I am also curious if there is plan to support true zero downtime. I know it's possible with nginx because it doesn't drop connections. Maybe that's an approach?

[matiasdecarli]

I'm running some tests, and even when I'm not seeing packets being lost, I'm seeing a high latency when scaling up/down the amount of containers behind a HAProxy. Suddenly some request start to take +3 seconds, and some more than 10.

[amir20]

Yes, I saw that too. Most of my calls were < 0.03 seconds and then suddenly jumped to 3+ seconds. I tried playing around with other proxies like Nginx. But I wasn't able to get it working with the time I had.

[lee5i3]

I am getting downtime as well

If I switch linked services, there is no downtime as long as I did not redeploy the service, but if I redeploy a linked service then point haproxy, the site goes down for approx 3-5 seconds

Everything I found so far, says that if HAProxy config reloads, the connections are dropped, I am hoping this changes, as our project is going to require a zero downtime in order for us to successfully use continuous deployment in our process

[gsdevme]

I am seeing downtime also while using siege, by stopping one of the containers causing a reload of ha-proxy

Transactions:                464 hits
Availability:              99.78 %
Elapsed time:              13.48 secs
Data transferred:               2.07 MB
Response time:              0.03 secs
Transaction rate:              34.42 trans/sec
Throughput:                 0.15 MB/sec
Concurrency:                1.07
Successful transactions:         464
Failed transactions:               1
Longest transaction:            5.01
Shortest transaction:           0.00

[johnnyhalife]

I got HA proxy not to drop traffic by using GitHub multibinder https://GitHub.com/johnnyhalife/docker-multibinder

[samuelgiles]

Multibinder looks like a great solution 👍

[dev-lusaja]

Hi
I make a bash hook for listen docker events and reload the balancer, see this example code

https://github.com/dev-lusaja/Docker-Cloud-HAproxy/tree/master/Compose-v2