Major performance issue with hapi.js 15.x

[lostthetrail]

What are you trying to achieve or the steps to reproduce?

Under higher concurrency Hapi 15.x memory usage spikes and fails, where Hapi 14.x had no problem

'use strict';

const Hapi = require('hapi');
const hoek = require('hoek');

const server = new Hapi.Server({
    connections: {
        router: {
            stripTrailingSlash: true
        }
    }
});

server.connection({
    port: process.env.NODE_PORT || 8080,
    host: process.env.HOSTNAME || '0.0.0.0',
    routes: {
        response: {
            emptyStatusCode: 204
        }
    }
});

server.route([
    {
        method: 'GET',
        path: '/validate',
        handler: function handle(request, reply) {
            return reply({
                validate: true
            });
        }
    }
]);

server.start(function handleServerStart(serverStartErr) {
    hoek.assert(!serverStartErr, serverStartErr);
    console.log(`Server running at: ${server.info.uri}`);
});

artillery quick --duration 60 --rate 100 -n 100 http://localhost:8080/validate

What was the result you received?

With Hapi 15.x, you will see a "JavaScript heap out of memory" and it will fail.
With Hapi 14.x, you will have a successful test and have zero errors.

What did you expect?

No major failures and zero errors.

Context

• node version: 6.5.0 & 4.5.0
• hapi version: 15.0.3
• os: OSX & Centos
• any other relevant information: I am aware this is a very high RPS to push through a server. But it is an attack vector. I also was able to replicate this with JMeter, so the load tool isn't important in this case. Just high concurrency and RPS.

[hueniverse]

I cannot reproduce this issue with hapi@15.0.3. I ran the artillery test about 20 times against the same process. I ran it 3 times in parallel. I then ran 3 parallel artillery processes at 200 rps over 2 minutes - I got lots of errors on the artillery side (timeouts and disconnected sockets) but not a single memory error on the hapi server side. When that test was done, I was able to connect to the server and get the response.

Can someone else reproduce this?

And to the people posting thumbs up, what does that mean in this context? That you can reproduce it? That you are just concerned? You are not being helpful in any way.

[mtharrison]

I can't replicate any memory issues but I do see slightly higher CPU usage with hapi 15.x.x compared with 14.x.x. The additional time is mainly spent in Joi validation for every request coming from internals.Podium.registerEvent

[genediazjr]

Hi, @mtharrison what did you use for the profiling?

[mtharrison]

V8 inspector (node --inspect).

[genediazjr]

@mtharrison thanks!

[lostthetrail]

It appears there is some correlation between the failure I am seeing in Hapi and the version of Artillery used during the test.

I have pushed up an example with the specific version I used pinned:
https://github.com/lostthetrail/hapi-15-mem-issue

npm start in terminal 1
npm test in terminal 2

I had a co-worker clone the repo and follow the README. He was able to reproduce via the repo above.

[Marsup]

The most important source of memory seems to be _notificationsQueue on the hapi Connection object. I don't see why it would retain so much unless we never reach a point where setImmediate runs those events, but I thought the whole point of this against nextTick was IO, I'm puzzled.

[Marsup]

@hueniverse doesn't seem to make a difference but is https://github.com/hapijs/podium/blob/master/lib/index.js#L168 really in the right place ?

[lostthetrail]

Let me know if there is anything else I can do.

[rfk]

FWIW, I can confirm that the steps in #3347 (comment) reproduced the bug for me as well, with the server process from npm start eventually dying with Allocation failed - process out of memory.

Based on #3347 (comment) I added some diagnostic logging inside podium, and the process got up to logging _notificationsQueue.length == 642928 before it died.

[Marsup]

That's a difficult trade-off, if we switch from setImmediate to nextTick to allow node to actually process those, we're accepting less sockets, but at least the server doesn't fall. I still find it weird that node is never ever running setImmediate as long as IO is keeping it busy, this looks like a flaw in node.

[DavidTPate]

@Marsup It actually makes sense (based on the docs) that what you are seeing is happening, below is why:

setImmediate

Schedules the "immediate" execution of the callback after I/O events' callbacks and before timers created using setTimeout() and setInterval() are triggered. Returns an Immediate for use with clearImmediate().

When multiple calls to setImmediate() are made, the callback functions are queued for execution in the order in which they are created. The entire callback queue is processed every event loop iteration. If an immediate timer is queued from inside an executing callback, that timer will not be triggered until the next event loop iteration.

nextTick

The process.nextTick() method adds the callback to the "next tick queue". Once the current turn of the event loop turn runs to completion, all callbacks currently in the next tick queue will be called.

This is not a simple alias to setTimeout(fn, 0), it's much more efficient. It runs before any additional I/O events (including timers) fire in subsequent ticks of the event loop.

So if we are talking about the processing then all of our callbacks made using setImmediate will be called after I/O completes and then called after the event loop completes, so in this case if I/O never completes they won't be called. Whereas those added with nextTick which will process once the current event loop completes and prior to other I/O.

[Marsup]

I know that but I'm having a difficult time understanding how IO can never leave any tiny amount of time for it to kick in.

[vladikoff]

Allocation profile chart below:

Download it via:

hapi15v3.heapprofile.zip

based on https://github.com/lostthetrail/hapi-15-mem-issue