Eliminate malloc_limit creepers

[evanphx]

Currently, the malloc_limit creeps up slowly on bigger heaps. This eliminates that by only increasing the limit if the malloc_increase is significantly larger than malloc_limit.

[SamSaffron]

I guess the thing that confuses me is that we have

RUBY_GC_MALLOC_LIMIT_MAX and RUBY_GC_OLDMALLOC_LIMIT_MAX now ... the big bug seems to be that they were not respected properly.

Once respected as in @ko1 s patch, is the creeper protection needed ?

[csfrancis]

I agree - couldn't this be resolved by using @ko1's patch and properly tuning RUBY_GC_MALLOC_LIMIT_MAX for the application?

[evanphx]

The problem I have with @ko1's patch is that it basically forces a FULL GC every 32M (by default) worth of allocations. This means that the full GC runs more often than it probably needs to, eliminating the whole reason for having generations in the first place.

[evanphx]

IMHO, the idea of a "MAX" value is wrong entirely. The system needs to be tuning it's behavior based on how the heap is used.

[SamSaffron]

@evanphx that was always the intention of max there, remember where we came from ... in Ruby 2.0 GC MALLOC LIMIT was a measly 8MB.

Agree we can eventually do better but I think just respecting the new max should be good enough to stop the majority of malloc creep pain out there. Which leave us only with the memory doubling issue.

32MB for Discourse is just enough not to be triggering malloc related GCs, though under Rails 3 minus a freedom patch or 2 it was not enough (probably 50-100MB)

32MB max may be slightly low as a default.

[csfrancis]

FWIW, less than 1% of Shopify's major GC's are triggered by breaching malloc limit.

[evanphx]

@csfrancis With our without @ko1's latest patch? Before his latest patch, the limit is pushed so high you'll hit it less and less as time goes on.

[csfrancis]

We're on 2.1.1 with RUBY_GC_MALLOC_LIMIT_MAX set to 128MB. We're also using the gctools OOBGC, which may be helping us out too. Here's a timechart of malloc_increase over the past 4 hours from one of our production app servers (Unicorn):

[mperham]

Memory bloat is a constant problem for our puma and sidekiq daemons. Generally monit has to restart them every day or so. We're on 2.1.1 with @SamSaffron's tunings.

[SamSaffron]

@mperham did you try @ko1's patch (its literally a 2 line change) also a https://github.com/ko1/gc_tracer dump will be a godsend debugging it.

need to figure out if its a malloc creep or heap bloat.

[mperham]

No, I don't run production on anything but stable releases.

On May 1, 2014, at 18:25, Sam notifications@github.com wrote:

@mperham did you try @ko1's patch (its literally a 2 line change) also a https://github.com/ko1/gc_tracer dump will be a godsend debugging it.

—
Reply to this email directly or view it on GitHub.

[SamSaffron]

Well you need to find out if this is malloc creep or heap bloat.

Only way to do that is by either running gc tracer or at least getting GC.stat from the process once memory creeps up. You can attach to it and get that using rbtrace if you include it.

[csfrancis]

Another option is to use the gc logger in https://github.com/tmm1/gctools. This outputs the gc trigger for every gc to stderr.

[k0kubun]

It seems to have a conflict now. Could you rebase this against master?

[evanphx]

This patch is older than my youngest child at this point. I have no clue if it works, the GC has changed a lot in 5 years.

[k0kubun]

This patch is older than my youngest child at this point.

I hope your child is also doing well 😄
Let me close this until you catch up the recent GC changes, but I also ping @ko1 to ask his opinion about this.