json parser taking close to 60 seconds during transaction

[justinpage]

Problem

Throughout the day we are seeing close to 60 second transaction calls from mobile app devices to our API. We've noticed in our new relic that the bottleneck appears to be the json parser middleware:

Looking at our code, we are simply adding the following package as a middleware to some of our available routes:

export const jsonParser = bodyParser.json({
    limit: "5mb"
});

We've identified that our apps seem to trigger this issue the most.

Reproducing

Sending the wrong content-length size in the header seems to reproduce the issue. We have a theory that the apps are calculating the content length automatically but are sending the wrong size.

Question

Is there any work around to this bottleneck issue? Something we can do in regards to not always expecting the correct content-length size? Or perhaps some max timeout limit in awaiting more data to come in? We noticed this issue mentioned a few years back that might be related?

https://github.com/stream-utils/raw-body/issues/32

Thanks!

[dougwilson]

Hi @KLVTZ in the end, the reading of the body (and notification of the end), is all done in Node.js core directly; for example, you should experience the same issues with the following code, not using the module at all:

app.use(function (req, res, next) {
  var str = ''
  req.on('data', (c) => str += c.toString())
  res.on('end', () => {
    req.body = JSON.parse(str)
    next()
  })
})

This is because Node.js core will not trigger the end event until it got all the bytes in the body, and the HTTP spec says that is determined by waiting for the number of bytes in the Content-Length header. If you're having clients either miscalculating the Content-Length or just stop sending bytes (drop off a cell network, for example), the only real recourse is to implement a timeout you want to wait around to read the request. Node.js does provide a mechanism to set the idle timeout on a socket (it defaults to 2 minutes).

If you have any thoughts on an implementation, please share, and even better would be to share as a pull request 👍

[justinpage]

@dougwilson thanks for the reply back with example of replicating this. We agree with your comments and ended up implementing a socket timeout based on our average requests in new relic. We also added a baked in timeout that construct a race condition, if the body cant be parsed in the allotted time then we return an error back to the client. This seems to have giving us better averages or at least tackled the timeout spikes we've seen in new relic.