Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Memory leak in log processing #2

Closed
roidrage opened this Issue Dec 29, 2012 · 7 comments

Comments

Projects
None yet
4 participants
Owner

roidrage commented Dec 29, 2012

Memory usage when processing logs regularly spikes up to more than one GB, although the heap size is restricted to 384M.

Given some previous dissections of the heap we did a while ago, Josh suggested this might be related to the notifications. Given the only async code that's still in there is using Pusher, it might be easy to reproduce this. Just fire an insane amount of async notifications, maybe have something on the other end to process them, and watch the memory usage.

Owner

roidrage commented Jan 28, 2013

@svenfuchs @drogus could one of you have a look into this? it's regularly bringing down gatekeeper and logs unfortunately.

Owner

svenfuchs commented Jan 28, 2013

Unfortunately not, no, but i've also lost track of this.

Our own notifications aren't being handled async any more since October (which I think was the time when @joshk reported the memory leakage with the local thread queue and blocks). Instead they're now running just inline, synchronously, as part of the normal code.

travis-ci/travis-core@6444c6e#L30L40

I believe the implementation of async/local hasn't changed since so it might still be leaking blocks (although "leak" might not be the right term). Afaik gatekeeper does not use this local queue any more, while logs does. So there might be other things to watch out for, too.

Owner

roidrage commented Jan 28, 2013

Could you please look into this then? You're most familiar with this part of the code. Would be great to get this issue fixed so we're not hunted by dynos running out of memory.

Owner

svenfuchs commented Jan 28, 2013

Will try.

Contributor

henrikhodne commented May 6, 2013

Is this still an issue?

Owner

roidrage commented May 6, 2013

Yes, travis-logs still has memory issues.

Owner

joshk commented Jul 6, 2013

Removed core, switched to sequel, things are way better, NOW TO WRITE TESTS!

@joshk joshk closed this Jul 6, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment