Support for pluggable proxy layer

[yagnik]

I was wondering what your views are in making the proxy layer pluggable which would allow me to use nginx instead of haproxy. I didn't want to prematurely start coding so wanted to hear your opinion first.
@igor47

[jolynch]

@yagnik I've been pretty interested in that for a while. I started down that path with the config generators refactor, but I didn't finish the job. There are only really two blockers:

1. There is still some coupling between haproxy as the output format and the service watchers, but it should be straightforward to remove.
2. We'd have to make it ok to have 'nginx' or w.e. at the top level instead of just haproxy

Neither of these is tough, but it is tough to say "this works" without running it in production, and while I can't speak for airbnb I can say that we (Yelp) are reasonably happy with haproxy as the core routing component now that we figured out how to restart it without dropping traffic.

[yagnik]

@jolynch what was your reason for using nginx instead of haproxy ?
The part that got me concerned is your post http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html which I believe is not a problem in nginx land cause process swapping doesn't happen. Is that the same reason you thought about using nginx ?

[jolynch]

@yagnik Yes we considered switching to nginx due to the reload issue, especially as PaaSTA matured and we found out that SmartStack would be restarting HAProxy basically all the time.

However, since our zero downtime reload strategy shipped 8 months ago and we invested serious effort in performance tuning our HAProxy configs we've had relatively few problems with HAProxy itself. We're trying to work with the HAProxy maintainers to complement their currently supported dynamic changing of hostnames (in 1.6) with dynamic changing of ports, which would mean that we could theoretically pre-populate and then swap in and out as we get backends, although I'd still be worried about some of the stuff we can only do on reload. On the other hand Linux 4.4 will ship with a sweet refactor to the TCP SYN handling stack which will fix the 3WHS bug in Linux (causes most of HAProxy's dropped packets with SO_REUSEPORT) and if Eric and co. can figure out this patch then HAProxy will be 100% hitless on a reload.

I guess what I'm trying to say is that we've spent a lot of effort finely tuning HAProxy for high availability and performance, so investing in nginx seems like a hard move for us especially when Linux might finally fix their SO_REUSEPORT implementation. It's worth noting that "switching to nginx" isn't a silver bullet. Nginx is still a relatively young load balancer (i.e. just recently gained the capability to load balance TCP in the open source version, last I checked healthchecks are still part of nginx+ only, etc ...), and there is no guarantee that it won't have some other show stopping problem.

All that being said, I'd love if there was a nginx output format for Synapse, it's just hard for me to find time to work on it, but I'd be happy to help in any way I can.

[yagnik]

thanks @jolynch sorry about the delayed reply. Also curious do you guys use chef/puppet for continuous deployment to all nodes for synapse/nerve ? How do you guys manage versioning in that case ?

In the mean time I'll pull away some time for getting nginx support, wait for my pr :)

[jolynch]

@yagnik Yea we basically build debian packages of synapse, nerve, synapse-tools and nerve-tools using Docker. We have some glue puppet modules that tie together a synapse and nerve puppet module which basically install the aforementioned packages and setup cron jobs to run the jobs from synapse-tools and nerve-tools.

One of these days we'll get our act together and make a "how to smartstack at scale" website or something with a single binary you can curl and pipe to ba^H^H... I mean a debian package or something.

[yagnik]

Never got around to doing this and we went ahead with haproxy too and now haproxy doesn't have a reload issue: https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ so happy with that.

As an aside we ended up building an rpm using omnibus: https://github.com/yagnik/omnibus-smartstack and use dendrite:https://github.com/yagnik/dendrite to build nerve and synapse config.

Going to close this issue for now.

[yagnik]

Thanks for all the help @jolynch !

[jolynch]

@yagnik oh also fwiw Synapse supports this now through config generators :-)

See the readme for how to do it and https://github.com/jolynch/synapse-nginx for an example of a proxy plugin.

[yagnik]

👍