Support load balancing with nginx dynamic upstreams

[subnetmarco]

Support for dynamic upstreams that will enable dynamic load balancing per API.

 # sample upstream block:
 upstream backend {
    server 127.0.0.1:12354;
    server 127.0.0.1:12355;
    server 127.0.0.1:12356 backup;
}

So we can proxy_pass like:

proxy_pass http://backend;

[bobrik]

I'd love to see that too. My use-case is routing requests to dynamic mesos tasks with zoidberg, kong would be a good candidate to do the routing part. I was going to use nginx with 2 ports anyway. Let me know if it makes sense to use kong for this.

Here are the options I see:

1. New Feature: added ngx_http_lua_upstream_set_peer_addr() openresty/lua-upstream-nginx-module#11 can be used to preallocate list of upstreams and only use some subset of that list
2. Hi,I pull a request for add_server to upstream,and include testing code openresty/lua-upstream-nginx-module#12 can be used for dynamic allocation of upstreams, but list can only grow, so previous PR is needed as well
3. https://twitter.com/agentzh/status/580170442150846464 — not public yet, needs manual handling of retry logic, but can replace both previous versions

Reloading nginx is not an option since it triggers graceful restart for all the worker processes. We use that mechanism with haproxy in marathoner, but long-lived sessions force previous instances of haproxy to stay alive for extended periods of time. Nginx is using more processes than haproxy, so it would be even worse. Deploying a lot can cause spinning thousands of proxying processes for no good reason.

[Tenzer]

Nginx can already do this in the Plus version: http://nginx.com/products/on-the-fly-reconfiguration/

[bobrik]

Yep, that's another option starting at $7000 annually for 5 servers.

[Tenzer]

You can also buy the license for one server at $1500 per year: http://nginx.com/products/pricing/. I'm not saying it's cheap or anything but it's an alternative if people wasn't aware of it.

[bobrik]

Another option: https://github.com/yzprofile/ngx_http_dyups_module

[bobrik]

Looks like dyups can do the trick: https://github.com/bobrik/zoidberg-nginx. Nginx config and lua scripts could give an idea how it works.

I'm not sure about stability of this thing, though.

[subnetmarco]

Just a quick update on this, this feature is important and we feel like an initial implementation should be done. It's not going to make it into 0.3.0 (currently working on implementing other major features, including SSL support and path-based routing), but it definitely should show up within the next two releases. We are trying to keep the release cycles very short, so it shouldn't take too much time.

Of course pull-requests are also welcome.

[bobrik]

Can you tell me how this is going to be implemented?

On Wednesday, May 20, 2015, Marco Palladino notifications@github.com
wrote:

Just a quick update on this, this feature is important and we feel like an
initial implementation should be done. It's not going to make it into
0.3.0 (currently working on implementing other major features, including
SSL support and path-based routing), but it definitely should show up
within the next two releases. We are trying to keep the release cycles very
short, so it shouldn't take too much time.

—
Reply to this email directly or view it on GitHub
#157 (comment).

Regards, Ian Babrou
http://bobrik.name http://twitter.com/ibobrik skype:i.babrou

[subnetmarco]

@bobrik As you pointed out in one of your links, apparently the creator of OpenResty is building a balancer_by_lua functionality which should do the job - so I am investigating this option.

The alternative is taking one of the existing pull requests on the lua-upstream module and contribute to them to make them acceptable and implement any missing feature we might need.

[bobrik]

@thefosk the only public info about balancer_by_lua is the tweet by @agentzh. Contributing to existing PRs to lua-upstream module also involves his approval :)

Take a look at ngx_http_dyups_module, this stuff reuses logic from nginx, as opposed to balancer_by_lua. It worked for me in my tests without crashes: 1-100 upstreams, updating every second with full gradual upstream list replace, 8k rps on 1 core with literally no lua code execution when serving user requests. Not sure about keepalive to upstreams, https and tcp upstreams, though.

[subnetmarco]

@bobrik yes, we will start working on this feature in the next releases, so we will monitor any announcement about balancer_by_lua in the meanwhile. If balancer_by_lua won't be released publicly during this time, then we will need to find another solution.

The requirement for Kong would be to dynamically create an upstream configuration from Lua, then dynamically populate the upstream object with servers and use it in the proxy_pass directive. Do you think we can invoke ngx_http_dyups_module functions directly from Lua bypassing its RESTful API?

The use case wouldn't be to update an existing upstream configuration, but to create a brand new from scratch, in pseudo-code:

set $upstream nil;
access_by_lua '
  local upstream = upstream:new()
  upstream.add_server("backend1.example.com", { weight = 5 })
  upstream.add_server("backend2.example.com:8080", { fail_timeout = 5, slow_start = 30 })
  ngx.var.upstream = upstream
';
proxy_pass http://$upstream

[bobrik]

@thefosk I create upstreams on the fly and update them on the fly with ngx_http_dyups_module. Moreover, I do it from lua code in RESTful API.

Take a look:

https://github.com/bobrik/zoidberg-nginx/blob/master/nginx.conf
https://github.com/bobrik/zoidberg-nginx/blob/master/zoidberg-state-handler.lua
https://github.com/bobrik/zoidberg-nginx/blob/master/zoidberg-proxy-rewrite.lua

Your pseudo-code implies that you create upstream on every request, I only do that on every upstream update. In zoidberg-nginx there is also some code for checking if upstream exists to prevent endless loops, but I found out that it is avoidable with this trick:

        location / {
            set $where where-am-i.zoidberg;
            proxy_pass http://$where;
        }

Upstream where-am-i.zoidberg is created on the fly and it's not a real domain name, no recurse proxying to itself until worker_connections are exhausted happens.

[subnetmarco]

@bobrik thank you, I will look into this

[Gingonic]

I'm also looking at an API manager that makes sense for mesos/marathon. After spending much time with my friend Google I came to the conclusion that right now there is only one option available : choose a service discovery (consul, haproxy brige, zoidberg,...) and add an API Proxy on top of it (Kong, Repose, Tyk, ApiAxle, WSO2 AM, etc.).
Frankly I don't see why I should put a proxy on front of a proxy. It would make a lot of sense to have a lightweight API manager+service discovery service in one piece of middleware. So +1 for this feature. What is the planing state?

[krishnaarava]

+1 for this feature

[ngbinh]

same here 👍

[neilalbrock]

Another +1 for me good sir

[agentzh]

The balancer_by_lua* directives from ngx_lua will get opensourced soon, in the next 3 months or so.

[subnetmarco]

@agentzh very good news, looking forward to trying it

[jdrake]

@thefosk +1

[sonicaghi]

+1

[bobrik]

@agentzh any chance to get TCP support in balancer_by_lua as well?

[agentzh]

@bobrik I'm not sure I understand that question. Are you talking about doing cosockets in the Lua code run by balancer_by_lua or you mean using balancer_by_lua in stream {} configuration blocks instead of http {} blocks?

[andy-zhangtao]

@Tieske Hi Tieske, what is Kong mgt API? And how to use POST /upstreams ? I didn't find reference in Kong document(v0.8) . I also need this feature, but I don't know how to do. :-(

[subnetmarco]

@andy-zhangtao this feature is currently being built. We are aiming to release it in the 0.10 version.

[andy-zhangtao]

@Tieske Got it! Thanks Tieske

[merlindorin]

@thefosk awesome!

We dream to build microservice that can be autoregistered against kong <3

[subnetmarco]

We dream to build microservice that can be autoregistered against kong <3

@iam-merlin this is exactly our vision. The next couple of releases will be very exciting for what it concerns microservices orchestration.

[merlindorin]

btw, @Tieske , if you want a beta tester or a feedback, don't hesitate to ping me. I've some code ready for that :D (I started to write an autoregister library... before I found that kong does not have this feature yet :'( )

[alza-bitz]

We dream to build microservice that can be autoregistered against kong <3

@iam-merlin @thefosk how about a Kong adapter for registrator? This could register Docker-based services with Kong, if the service has the environment SERVICE_TAGS=kong for example.

n.b. this wouldn't replace other registrator adapters e.g. the Consul adapter, it would compliment them (i.e. could be used in combination, or just use the Kong adapter on it's own).

I am planning to deploy Kong for a work project when it supports SRV records fully, in the meantime I have a nginx Docker container which dynamically adds upstreams based on SERVICE_TAGS=nginx.

[merlindorin]

@alzadude , I don't know registrator but from my point of view (and what I read), registrator needs to be ran on each host... with a feature like this issue, you don't need docker (and consul or other service registry), just a Plain Old Request (^^) and maybe we will have health check after (in another plugin).

From my point of view, Consul is very heavy for microservice... I really like the simplicity with Kong (and no dependencies) but right now... doing a microservice architecture with Kong and without a service registry is a pain.

Anyway, I don't think my point of view is relevant... I'm just a user of kong, not the best one and not a lua developer :P xD.

[subnetmarco]

Just so you know, once @Tieske finishes this implementation, I will provide some sample Docker-Compose templates that show how to use Kong with a microservice orchestration pattern.

I do personally like ContainerPilot more than registrator though.

We will also be able to provide a pattern that doesn't involve having any third-party service discovery in place because, effectively, Kong's new /upstreams will become the service discovery layer.

[Tieske]

A PR with intermediate status is available now (see #1541)

Any input and testing is highly appreciated, see #1541 (comment)

so if anyone else wants to test like @iam-merlin, please check it out.

[merlindorin]

@Tieske it works xD

I've made some tests and it seems to works as expected (just the dns part, I didn't test yet balancer_by_lua).

It seems dns is missing in your requirement (I'm not a lua dev) and I've to installed it manually (I got on error at the first start and I install https://github.com/Mashape/dns.lua).

Do you have any documentation about your code or this will be ready later?

[Tieske]

@iam-merlin thx for testing 👍. Docs will be later, as it might still change.

[Tieske]

besides #1541 (internal dns) there is now #1735 which implements the upstreams feature discussed above (1735 builds on top of 1541).

testing is once again highly appreciated!

[tomdavidson]

Cant not deploy load balancing with round robin. We need least open connections or fastest response time - anything but round robin.

[Tieske]

'least open connections' does not make sense in a Kong cluster. 'response time' is being considered, but not prioritized yet.

[Tieske]

closing this as #1735 has been merged into the next branch for the upcoming release.

[avanathan]

How to implement keep_alive without upstream? Because upstream doesn't support dynamic IP resolution and we cannot use keep_alive outside of upstream.