New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce Sidekiq memory usage 2-4x #7257

Closed
mperham opened this Issue Apr 24, 2018 · 8 comments

Comments

Projects
None yet
5 participants
@mperham
Copy link

mperham commented Apr 24, 2018

Hi, Sidekiq author here. I see a lot of complaints from people about Mastodon's Sidekiq memory usage. I'd recommend you read this blog post and switch to jemalloc in your Docker image. I've heard from a dozen users that the memory reduction is "miraculous". You install jemalloc and then use LD_PRELOAD to override the default glibc malloc when executing Ruby.

https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
https://github.com/jemalloc/jemalloc/wiki/Getting-Started

@Gargron

This comment has been minimized.

Copy link
Member

Gargron commented Apr 24, 2018

@mperham Hey! I'm a big fan of Sidekiq. However, it's not so simple. We use Alpine Linux in Docker, and apparently jemalloc does not work well on Alpine: https://bugs.ruby-lang.org/issues/13524

@mperham

This comment has been minimized.

Copy link
Author

mperham commented Apr 24, 2018

@Gargron It appears to be fixed with Alpine 3.7 and jemalloc 5.0.1. I could repro the segfault before, can't with those versions.

@Gargron

This comment has been minimized.

Copy link
Member

Gargron commented Apr 24, 2018

Hmm, we had to go back from 3.7 to 3.6 because 3.7 introduced random segfaults into Sidekiq! (#6734) 😅

@mperham

This comment has been minimized.

Copy link
Author

mperham commented Apr 24, 2018

Oy. I would bump to 2.4.4 first before giving up. It's common for MRI to get fixes for this type of thing and 2.3.0 is several years old.

@saper

This comment has been minimized.

Copy link
Contributor

saper commented Apr 24, 2018

Please take into consideration that Alpine has a very small default thread stack size (https://wiki.musl-libc.org/functional-differences-from-glibc.html#Thread-stack-size voidlinux/void-packages#4147).

A story with RSpec:

https://www.madetech.com/blog/a-tale-in-adopting-alpine-linux-for-docker-problems-we-faced-with-rspec-testing

libuv (and therefore node) avoid the problem since node 8.x by increasing the stack size explicitly https://github.com/libuv/libuv/pull/1526/files#diff-391600498c36f644bc9439a0b05a12f7R194

It is possible to apply a quick work around with LD_PRELOAD (https://github.com/jubel-han/dockerfiles/blob/master/node/Dockerfile) and see if that helps.

@mperham

This comment has been minimized.

Copy link
Author

mperham commented May 1, 2018

Since you aren't using glibc, it's not clear to me that memory bloat is a problem. glibc has terrible bloat but Alpine uses musl libc which I assume has a different allocator. Closing.

@mperham mperham closed this May 1, 2018

@nightpool

This comment has been minimized.

Copy link
Collaborator

nightpool commented May 1, 2018

I'm pretty sure we still see memory leaks in sidekiq under docker but I can't speak to it myself, i run mastodon baremetal.

(not saying this is your fault, just saying it's maybe still good to look into the jemalloc thing)

maybe @0xa could give some more insight here? I know they OOM kill docker containers pretty often on mastodon.social. unsure about sidekiq specifically.

@mperham

This comment has been minimized.

Copy link
Author

mperham commented May 1, 2018

I've pitched the idea to have Ruby use jemalloc by default. Like any major change to a large project, it's a steep uphill climb, especially when we can't easily reproduce the problem. I'm going to look into building a custom Rails app which will reproduce the bloat and show that jemalloc helps it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment