Error R14 (Memory quota exceeded) on _web_ dyno

[eddierubeiz]

First over the weekend of Sept 23-24.. This issue is to track these errors.

[jrochkind]

we get R14 errors in scheduler/worker all the time, which is also annoying, but different.

Our web dyno actually has a lot of RAM, this is disturbing and unfortunate. But I think it would take a lot of time investigating to have anything at all to say on it (probably still not conclusive), and I don't at present plan to spend today doing that.

Perhaps just hope it doens't happen again?

One easy thing we can do is try implemetning the jemalloc patch again, something some people find reduces Rails memory consumption, including on heroku (need to google for references/links on this subject).

[eddierubeiz]

Jemalloc work from last fall: #1797

[eddierubeiz]

(from the web dyno memory use dashboard)

Roughly speaking, memory usage was hovering between 2 and 2.5 GB for most of Thursday through Sunday of last week.

[eddierubeiz]

Heroku hasn't reported any problems in the past week.

[jrochkind]

Happened again Monday Oct 2 between 7am and 9am.

It would be interesting to see if we had more traffic than usual then. Our rate-limiter didn't seem to catch anything during that period.

Nothing obvious in Heroku metrics either.

[jrochkind]

Our heroku production is a single performanc-m dyno, which has 2.5GB of RAM.

We currently run 5 puma processes on it, each with 5 threads.

So I guess our puma processes have edged over an average of 512MB/per, at least in some cases.

We could first try pemalloc.

We could reduce to 4 puma processes, reducing our capacity by 20%.

Or we could upgrade our dyno, however the next step up is a performance-l dyno, at an additional $250/month. However, that would give us significantly more capacity, it more than doubles our capacity (14GB RAM instead of 2.5GB, and around 4x the CPU capacity).

[jrochkind]

R14's continued this morning Monday May 2 past 9am as well, and may be ongoing.

I'm going to try deploying the jemalloc thing....

heroku buildpacks:add --index 1 https://github.com/gaffneyc/heroku-buildpack-jemalloc.git -r production
heroku config:set JEMALLOC_ENABLED=true -r production

Then deploy. Let's see how the graph looks after this.

If we keep this, make sure to document in wiki on heroku config.

[eddierubeiz]

Not sure if this is helpful but Scout does have some memory-related diagnostics.

[jrochkind]

The last R14 today error was 10:53:33AM Oct 2. So it's possible our jemalloc change improved things; or just the dyno restart that happened when we deployed.

We'll have to keep keeping an eye on things, will leave this ticket open in "in progress" for a bit to remind us to monitor.

[jrochkind]

The past week of jemalloc does seem to have on average saved 300-500MB of RAM -- and there have been no R14 errors in the past week.

We're going to say this is a real improvement, sufficient for now, close this ticket. Can be re-opeend or new ticket if problem re-arises.