Node fails quickly with std::bad_alloc yet there is plenty of unused heap size #1544
Description
- Node.js Version: 8.11.4
- OS: Linux (Amazon)
- Scope (install, code, runtime, meta, other?): runtime
We have a Node app that gets about 200 req / minute, mostly fetching data from Redis, JSON.parseing it and returning. It crashes reliably within 3-5 min of start with either
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
or
# Fatal error in , line 0
# Out of memory: HashMap::Initialize
The weird thing is that it uses only a part of its heap (provided our monitoring process.memoryUsage() isn't missing something) - we run it with --max-old-space-size=3072 (on a 4GB machine) and heapUsed < 750MB, heapTotal < 1GB, rss < 1GB, osMemoryFree 2 - 3 GB.
These memory crashes and lower rss/heap usage started after a deployment of a recent change so it is likely the change causes this, but
- Why is Node crashing with out-of-memory when there seems to be plenty of it available?
- What are we missing in our monitoring? Could it be that it is running out of space in same weird kind of memory, other than the standard heap, that our monitoring is not watching?
- Can we supply some command-line options to get more insight into which memory it is lacking / where is it crashing?
Thank you so much!
This is the code that was deployed when memory usage dropped and the OOM crashes started:
/** Allow max one promise 'in progress' for the given `key`; return it if requested again before finished. */
class PromiseThrottle {
constructor() {
this.requestsInProgress = {};
}
throttle(key, promiseMaker) {
if (this.requestsInProgress[key]) return this.requestsInProgress[key]; // Return the in-progress promise
return this.requestsInProgress[key] = new Promise((resolve) =>
resolve(timeout(promiseMaker(), time.fifteenMinutesInMs)))
.then(d => { delete this.requestsInProgress[key]; return d; })
.catch(err => { delete this.requestsInProgress[key]; throw err; });
}
}
const redisGetThrottle = new PromiseThrottle();
/** Get a key - return null when not present. */
function get(key) {
return redisGetThrottle.throttle(key, () => redis.get(key).then(JSON.parse));
// BEFORE this change it was just:
// return redis.get(key).then(JSON.parse);
}