Add support for node cluster

[jfrconley]

Adds support for node cluster module using the SO_REUSEPORT option for the uv socket. This can be disabled with the server option reusePort, which defaults to on. SO_REUSEADDR is used instead on windows.

[roblav96]

@jfrconley Awesome contribution here! Would love to see this merged =] @mafintosh

[gakada]

Is this a Linux only solution?

AFAIK SO_REUSEPORT is quite special in Linux (and DragonFly BSD), on other systems using cluster with n workers and reusePort might just create n - 1 useless workers, at least I tried running this test, Linux version is working as expected (different worker PIDs), while on Windows it looks like just one worker (same PID, maybe "stealing" vs actual load balancing, though I'm not sure what is the reason why cluster doesn't work with turbo-net/turbo-http how it works with net/http).

On Linux it does look nice though

$ node benchmarks/compare/turbo-http.js

$ wrk -t10 -c500 -d10s http://127.0.0.1:5050/
Running 10s test @ http://127.0.0.1:5050/
  10 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     5.22ms    3.34ms 216.04ms   98.10%
    Req/Sec     9.45k     2.18k   60.56k    92.12%
  943474 requests in 10.10s, 71.08MB read
Requests/sec:  93420.25
Transfer/sec:      7.04MB

$ node benchmarks/compare/turbo-http-cluster.js

$ wrk -t10 -c500 -d10s http://127.0.0.1:5050/
Running 10s test @ http://127.0.0.1:5050/
  10 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.41ms    1.96ms  32.24ms   88.00%
    Req/Sec    56.80k    25.85k  146.52k    60.96%
  5668663 requests in 10.08s, 427.08MB read
Requests/sec: 562514.13
Transfer/sec:     42.38MB

[jfrconley]

@gakada You are correct in that on Windows, you get effectively a single worker as ports cannot be shared. SO_REUSEADDR (at least on Windows) essentially steals the port from whoever last had it. It is only included for compatibility reasons (you shouldn't be deploying a node server on Windows anyways). This is similar to what node does by default; All platforms other than Windows use round-robin with SO_REUSEPORT and Windows distributes connections to workers through the master. You could implement something similar on top of this, but the gains would be middling and likely not worth the effort.

edit: You'll also get higher numbers if you limit your number of threads to your number of cores (or double if you have hyper threading)

edit 2: add context

[gakada]

Well I tried running

const cluster = require("cluster");
const http = require("turbo-http");

if (cluster.isMaster) {
  for (let i = 0; i < 2; ++i) {
    cluster.fork();
  }
} else {
  const pid = Buffer.from(`${process.pid}`);
  http.createServer((_, res) => res.end(pid)).listen(8080);
}

and it works on Linux (2 workers utilized), but doesn't work on Windows (only 1 worker).

Maybe we can have a test demonstrating that cluster load balancing is working (> 1 workers)?

[jfrconley]

@gakada Sorry, updated my response while you were posting yours

[gakada]

If I replace turbo-http -> http in that example it is using more than one worker on Windows.

[jfrconley]

@gakada Right, I address this in my response. Node uses a single master process on windows that handles all connection termination, but then uses an IPC to distribute them to workers. Again, you could do this on top of turbo-http, but I'm not convinced the benefits would outweigh the costs

[gakada]

Btw, it is the same "not working" behavior on OS X. So I guess what I was pointing out is that if you test vanilla http + cluster it will work as expected on all systems (Linux, Windows, OS X, *BSD), while in this case it is only meaningful in Linux (at least not on Windows and OS X, probably not on *BSD as well, [1], [2], [3]).

[jfrconley]

You're totally right, I done goofed. Upon further research it seems that the load distribution features of SO_REUSEPORT are exclusive to Linux and that server.listen in Node actually passes off most work to the master process (setting up file descriptors to listen on and such). More work will have to be done on properly integrating such logic here.