performance

[manast]

I would like to open a thread regarding performance, because I think it requires a bit more of attention than it has actually received lately.

Although this project does not aim to be as performant as other leading proxies such as HAProxy or nginx, I think most of its users would certainly be happy if the proxy does not degrade the performance to a nodejs server by a factor of 10x or 15x.

This closed issue: #929 shows that it is very easy to actually verify the performance degradation produced by the proxy. The workaround is to send a http.Agent with keepAlive: true, and maybe other finetunings. The issue refers to a FAQ, but I cannot find any FAQ. Also it would be great to provide a couple of things regarding http.Agent: 1) what is the tradeoff of using it? (if none, why is it not enabled by default), 2) Which is the optimal set of options for it? just changing arbitrarily and re-testing does not seem like a good approach, and also a dangerous one I may add, since the user most probably does not know what he is doing. 3) Why is http.Agent required to begin with?

One thing that may make many wondering is how is it possible that if a dummy http server in node js is capable of delivering responses in the order of magnitud 10k, a simple proxy infront of it, that should, in its most basic conceptual form, just pipe the data from source to target, reduces the performance to order of magnitude 0.5k. One could accept 50% degration, but not this much. Thing is, this may not be related to http-proxy at all, for instance I wrote this super simple example:

var http = require('http');
var request = require('request');

http.createServer(function(req, res){
  request.get('http://127.0.0.1:3000').pipe(res);
}).listen(8080);

http.createServer(function(req, res){
  res.writeHead(200);
  res.write('HELLO WORLD');
  res.end();
}).listen(3000);

And got this results (in node 4.5):

$ wrk -c 64 -d 15s http://localhost:3000
Running 15s test @ http://localhost:3000
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.85ms    3.41ms 307.55ms   99.38%
    Req/Sec     6.73k   505.00     7.10k    93.38%
  202128 requests in 15.10s, 24.87MB read
Requests/sec:  13384.99
Transfer/sec:      1.65MB

$ wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    55.45ms   12.50ms 158.42ms   88.76%
    Req/Sec   569.33    128.19   760.00     81.94%
  8265 requests in 15.07s, 0.98MB read
Requests/sec:    548.31
Transfer/sec:     66.40KB

So can it really be node's streams are so amazingly slow? It's a bit worrisome I must admit. Anybody as any insights that he wouldn´t mind to share?

[manast]

I did two more experiments that I find somehow interesting.

Using minimum http-proxy(I already got the same figures using https://github.com/OptimalBits/redbird based on http-proxy):

var httpProxy = require('http-proxy');
httpProxy.createProxyServer({target:'http://localhost:3000'}).listen(8080);

Gives basically the same result as using request streaming:

$ wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    52.03ms   10.90ms 119.16ms   87.05%
    Req/Sec   607.21    104.95   747.00     79.41%
  8257 requests in 15.04s, 0.98MB read
Requests/sec:    548.97
Transfer/sec:     66.48KB

On the other hand, using req-fast (https://github.com/Tjatse/req-fast) I get consistently this:

var reqFast = require('req-fast');
var http = require('http');

http.createServer(function(req, res){
  reqFast('http://127.0.0.1:3000').pipe(res);
}).listen(8080);

$ wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    49.73ms    9.96ms 142.05ms   89.36%
    Req/Sec   648.36    136.26     0.97k    72.80%
  16320 requests in 15.03s, 2.01MB read
Requests/sec:   1086.16
Transfer/sec:    136.83KB

So about twice as fast as request and http-proxy. Meaning that it is possible to implement a streaming proxy that is at least that fast inside http-proxy.

[manast]

Using keepAlive: true on the http.Agent with http-proxy:

wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    27.49ms    4.07ms 118.95ms   94.56%
    Req/Sec     1.17k   126.61     1.43k    88.67%
  35017 requests in 15.03s, 4.31MB read
Requests/sec:   2330.04
Transfer/sec:    293.53KB

Using keepAlive with req-fast:

var keepAliveAgent = new http.Agent({ keepAlive: true });
http.globalAgent = keepAliveAgent;

$wrk -c 64 -d 15s http://localhost:8080
Running 15s test @ http://localhost:8080
  2 threads and 64 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.62ms    7.44ms 102.17ms   85.51%
    Req/Sec   656.12     98.75     0.94k    79.84%
  16294 requests in 15.08s, 2.00MB read
Requests/sec:   1080.32
Transfer/sec:    136.09KB

So http-proxy gets a huge 4 times better performance with keepAlive, while req-fast stays the same.

[manast]

Previous experiments were aimed to just check what is the overhead for a minimal web server, now check this results when serving strings of different sizes from 32 bytes to 256Kb.

Test code:

var http = require('http');

var httpProxy = require('http-proxy');
var keepAliveAgent = new http.Agent({ keepAlive: true, maxSockets: 1000 });

var randomstring = require("randomstring");
var msg = randomstring.generate(2*1024);

httpProxy.createProxyServer({target:'http://localhost:3000', agent: keepAliveAgent}).listen(8080);

http.createServer(function(req, res){
  res.writeHead(200);
  res.write(msg);
  res.end();
}).listen(3000);

In this case I used needle for implementing a proxy since it gave best performance than req-fast and request:

var needle = require('needle');

http.createServer(function(req, res){
  needle.request('get', 'http://127.0.0.1:3000', null, {agent: keepAliveAgent, connection: 'keep-alive'}).pipe(res);
}).listen(8080);

And the results:

For me, the interesting here are basically 3 things:

• There is a lot of overhead to setup the streaming.
• For strings of size 32 and above, the proxy overhead is negligible.
• The streaming gives best performance with strings of size 32Kb, after that it starts degrading, which is strange, I was expecting less and less overhead so more and more raw throughput with large strings.

[manast]

I filled the gaps to get a more linear chart:

[dtjohnson]

I would definitely like to see this addressed as well. I've been using Nginx as a dynamic reverse proxy for some time now. It's worked well, but the Nginx configuration spec is massive and confusing. And the dynamic part requires scripting in Lua, which isn't the easiest. I'd like to implement some more sophisticated features to my proxy, but that's painful in Nginx/Lua. It would be much simpler to do in Node.js, but I'm running into the same performance issues as manast with this module.

I did my own benchmarking. I spun up a Docker container first with a simple Nginx server serving static text (because I knew it would be lightning fast). I spun up a second Nginx container set up as a reverse proxy for the first to use as a comparison. I then spun up a Node.js 6.3 container testing a proxying with this module and the built in Node http client as well as serving static content for comparison. I benchmarked with wrk using 10 and 100 connections. Then I repeated the whole process with a Node.js source (instead of the Nginx one) introducing a 100 ms delay before serving content.

Here is my proxy code:

const http = require('http');
const httpProxy = require('http-proxy');
const keepAliveAgent = new http.Agent({ keepAlive: true });

// Plain HTTP server w/ static content.
http.createServer((req, res) => {
    res.statusCode = 200;
    res.setHeader('Content-Type', 'text/plain');
    res.end('Hello World\n');
}).listen(8000);

// http-proxy w/ default global agent
const defaultAgentProxy = httpProxy.createProxy();
http.createServer((req, res) => {
    defaultAgentProxy.web(req, res, {
        target: "http://10.224.51.210:8080"
    });
}).listen(9000);

// http-proxy w/ keep-alive agent
const keepAliveAgentProxy = httpProxy.createProxy({ agent: keepAliveAgent });
http.createServer((req, res) => {
    keepAliveAgentProxy.web(req, res, {
        target: "http://10.224.51.210:8080"
    });
}).listen(9001);

// http-proxy w/ no agent
const noAgentProxy = httpProxy.createProxy({ agent: false });
http.createServer((req, res) => {
    noAgentProxy.web(req, res, {
        target: "http://10.224.51.210:8080"
    });
}).listen(9002);

// Node http client w/ default global agent
http.createServer((req, res) => {
    http.get({
        hostname: '10.224.51.210',
        port: 8080,
        path: '/'
    }, upstreamRes => {
        upstreamRes.pipe(res);
    });
}).listen(10000);

// Node http client w/ keep-alive agent
http.createServer((req, res) => {
    http.get({
        hostname: '10.224.51.210',
        port: 8080,
        path: '/',
        agent: keepAliveAgent
    }, upstreamRes => {
        upstreamRes.pipe(res);
    });
}).listen(10001);

// Node http client w/ no agent
http.createServer((req, res) => {
    http.get({
        hostname: '10.224.51.210',
        port: 8080,
        path: '/',
        agent: false
    }, upstreamRes => {
        upstreamRes.pipe(res);
    });
}).listen(10002);

And here were the results:

The Nginx and Node.js static results are just there as a theoretical floor. The proxy server can't possible be faster than just serving static content. Nginx was definitely faster, but not quite as fast as I thought. It does perform better with 100 connections, but that's probably just because Node is using a single thread and Nginx is using 2 (one per CPU on my server).

The Nginx proxy results are the real target. If we can get close to the performance of Nginx we're in great shape.

With the node-http-proxy module, we see a 2x latency and 1/2 the requests compared to Nginx. I didn't see any difference between the default global agent (no keep-alive and infinity sockets) and no agent at all. Unexpectedly, when I enabled keep-alive the latency quadruples for 10 connections. For the 100 connections it performed much better.

I also proxied using the built-in http client to make a request to the upstream source and piped the results into the response. I did this with and without keep-alive. The results with keep-alive were fantastic! Even with a single thread the response times were lightning fast--even faster than the Nginx proxy--in all scenarios.

So what am I missing? Why is there such a huge latency added by node-http-proxy?

[dtjohnson]

I just did a little more benchmarking. This time I put the source Node.JS container on one server, the proxy containers on a second server, and the wrk container on a third to get a more representative benchmarking and to avoid any competition of resources. I also added HAProxy to the mix and went up to 1000 connections. I also added a test with the Node.JS http client using cluster so there are 2 Node.JS workers (one per CPU). These are all with Node.JS 6.5. Here were the results:

Here "Direct" means direct access to the source server with no proxy. There's a lot of network fluctuation, but it looks like the direction connection, HAProxy, and the Node.JS HTTP client piping with cluster are all about equally performant. node-http-proxy, by comparison, performs horribly. :( I was also surprised to see how poorly Nginx performed.

Here's my nginx.conf

worker_processes auto;

events {
    worker_connections 1024;
}

http {
    server {
        listen 8080;
        location / {
            proxy_pass http://source.mydomain.com:8080;
            proxy_http_version 1.1;
            proxy_set_header Connection "";
        }
    }
}

Here's my haproxy.cfg

global
    maxconn 1024

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    bind *:8080
    default_backend default-server

backend default-server
    option http-keep-alive
    server s0 source.mydomain.com:8080

Here's my node-http-proxy JS code

const http = require('http');
const httpProxy = require('http-proxy');
const proxy = httpProxy.createProxy();
http.createServer((req, res) => {
    proxy.web(req, res, {
        target: "http://source.mydomain.com:8080"
    });
}).listen(8080);

Finally, here is my custom Node.JS HTTP client piping proxy with cluster (not sure if I got the error/abort handling done correctly though):

const cluster = require('cluster');
const http = require('http');

const numCPUs = require('os').cpus().length;
const keepAliveAgent = new http.Agent({ keepAlive: true });

if (cluster.isMaster) {
    for (var i = 0; i < numCPUs; i++) {
        cluster.fork();
    }
} else {
    http.createServer((req, res) => {
        const proxyReq = http.request({
            method: req.method,
            path: req.url,
            headers: req.headers,
            hostname: 'source.mydomain.com',
            port: 8080,
            agent: keepAliveAgent
        }, proxyRes => {
            res.writeHead(proxyRes.statusCode);
            proxyRes.pipe(res);
        });

        req.pipe(proxyReq);

        proxyReq.on("error", e => {
            res.write(e.message);
            res.end();
        });

        req.on("error", e => {
            proxyReq.abort();
            res.write(e.message);
            res.end();
        });

        req.on('aborted', function () {
            proxyReq.abort();
        });
    }).listen(8080);
}

[manast]

Quite interesting results. But how large is the data being proxied? as you can see in my results about it has a lot impact in the results. In any case, it will be interesting to know why http-proxy is performing so bad, I have to check the sources but I guess it is using http.request internally...

[dtjohnson]

Following your suggestion, I ran the benchmarks with a variety of message lengths (from 1B to 1MB) using the randomstring module like you did. I didn't see the same issue as you. I wonder if you are seeing some strange behavior by running your proxy server and your upstream source on the same single Node.js thread. The results were very interesting though. Here are the results with the same 3 server configuration but with an even faster upstream source.

Here is 10 connections:

And zoomed in on the <= 1kB messages:

Now 100 connections:

And zoomed in on the <= 1kB messages:

Now 1000 connections:

And zoomed in on the <= 1kB messages:

HAProxy performs very well, adding only a small overhead in all scenarios. Node.JS client piping makes a decent showing. Nginx performs surprisingly poorly, and node-http-proxy lags far behind. node-http-proxy failed completely in most of the 1000 connection runs.

[manast]

I guess more people is needed to verify this results, but if so, this means that is not unrealistic to say that we could have a nodejs based proxy that is competitive enough to be used for high traffic sites. Its still quite amazing that nginx performs so bad in these tests, I wonder if there is not something in the configuration or the setup that makes it perform so bad (disclaimer: I am a novice in nginx).

[manast]

Btw, it would be also highly relevant to test with HTTPS. We should agree that any serious site should work using HTTPS anyway, so that is the performance we should care about :)

[dtjohnson]

I came to the same conclusion. Despite whatever is going on with node-http-proxy and Nginx, it certainly seems a Node.js-based proxy is a realistic option.

I'm stumped as to what is wrong with Nginx. I have to believe it is some configuration issue. I tried disabling proxy buffering and caches--no help. I tried using IP address instead of DNS for the target--no difference. I disabled keep-alive (which I need to support server-sent-events) and that did improve performance but still well below the the performance of Node.js.

HTTPS is a good idea to test too. I tend not to think about that as in my setup my proxies are fronted by an AWS Elastic Load Balancer that does SSL termination. Would be interesting to see any differences there though. Gzip on the proxy would interesting too.

One other point of consideration is that Nginx and HAProxy in my tests aren't set up with scripting. They are just fixed proxying. In the Node.js proxy we have access to the entire JS language and libraries. To do a fair test with Nginx and HAProxy we'd have to test dipping into a Lua script execution on each request to do a fair comparison.

I'd love to script the entire benchmarking workflow so we can iterate on this more quickly and consistently. So dynamically spin up 3 servers, start the proxy servers, benchmark them, and then terminate the servers.

[manast]

As I mentioned above I am the author of https://github.com/OptimalBits/redbird, and I am using http-proxy for the actual proxy work, but maybe it is not so difficult to implement a http.request based proxy instead. The risk is that there are probably a lot of cases, common as well as edge, regarding handling of headers and other stuffs that are not so easy to get right without a long period of battle testing such as http-proxy already has have.

[dtjohnson]

Agreed. It's the variety of cases I worry about too, but I'm afraid it's a bridge I may have to cross. I don't think I'll be able to evolve my current dynamic Nginx proxy easily enough (especially seeing the Nginx performance issues from the above benchmarking). I really think I need to swtich to a Node.js based one.

Would love to see the authors of node-http-proxy chime in. I feel like I must be doing something majorly incorrect that would explain the performance issues...

[eezing]

@manast @dtjohnson Have you taken a look at #614? Providing node-http-proxy an https agent and setting maxScockets may help. There also appears to be potential slowdown from DNS when setting target other than ip address. May not make a difference in Node v4+