Sidekiq and puma processes increase memory usage over time

[nolanlawson]

Expected behaviour

Memory usage does not slowly increase over time

Actual behaviour

Sidekiq and puma processes both appear to leak memory

Steps to reproduce the problem

1. Run Mastodon using Ruby with jemalloc
2. Run free -m (on a Linux machine) to see available memory
3. Notice that the used column slowly increases over time

Apologies for the lack of details on this one. I'm not sure if you can actually repro it in a dev server; you might need a live Mastodon instance with actual people using it.

Just to put some numbers on this, I host two small instances on the same machine. When I restart all Mastodon processes, the used memory is around 1.8GB. But within a few hours, this will slowly start to creep up, eventually reaching 4GB (which is the limit on this particular machine). This means that a sudden need to run ffmpeg, a tootctl script, or a Postgres backup script can bring the server down, since there's no memory left.

Using top, it seems like the Sidekiq processes consume the most memory, followed by the Puma processes. Both seem to increase in memory usage over time, but Sidekiq grows faster.

The workaround I've adopted is to have this in the systemd config:

Restart=always
RuntimeMaxSec=21600

I.e. restart the process every 6 hours. (I'm thinking of bumping it to 3 hours though, since the memory creeps up that quickly.)

Perhaps another solution would be something like sidekiq-worker-killer? Assuming the source of the leak can't be found.

Specifications

3.3.0

[nolanlawson]

BTW just to be clear: it may very well be that Puma and Sidekiq eventually reach a plateau where no more memory is consumed. Based on my observations, it seems like the memory usage grows quickly at first, then more and more slowly over time.

Also, I used to have a machine with 8GB of memory, and back then I ran out of memory less frequently.

That said, if my Mastodon instances can run fine with ~2GB, I would love for them to continue using around that much. That way, I wouldn't need to upgrade to a machine with 8GB or more, just to accommodate the gradual memory growth. 🙂

[angristan]

Have you tried using MemoryMax= in order to limit the memory usage of the service itself?

[nolanlawson]

I haven't! Has this worked for you? Does it work with the Restart=always config?

[angristan]

That's a good question. Currently, I only use CPUQuota to prevent a single mastodon service (e.g.mastodon-sidekiq-default) to suck up all the CPU when a ffmpeg process is running (or similar).

I'm pretty sure I used MemoryMax in the past when I had a memory leak issue with Caddy, but I don't remember exactly what happened.

My test instance is on a 2 GB RAM server, and Mastodon get OOM killed all the time:

root@momo ~# dmesg -T | grep 'Out of memory'
[Thu Jun  3 09:30:12 2021] Out of memory: Kill process 26877 (bundle) score 124 or sacrifice child
[Thu Jun  3 09:52:27 2021] Out of memory: Kill process 27029 (bundle) score 124 or sacrifice child
[Thu Jun  3 10:21:32 2021] Out of memory: Kill process 27235 (bundle) score 123 or sacrifice child
[Thu Jun  3 10:33:03 2021] Out of memory: Kill process 27711 (bundle) score 125 or sacrifice child
[Thu Jun  3 10:38:42 2021] Out of memory: Kill process 27961 (bundle) score 124 or sacrifice child
[Thu Jun  3 10:44:24 2021] Out of memory: Kill process 28026 (bundle) score 123 or sacrifice child
[Thu Jun  3 10:53:16 2021] Out of memory: Kill process 28126 (bundle) score 125 or sacrifice child
[Thu Jun  3 11:09:43 2021] Out of memory: Kill process 28344 (bundle) score 124 or sacrifice child
[Thu Jun  3 11:18:09 2021] Out of memory: Kill process 28577 (bundle) score 125 or sacrifice child
[Thu Jun  3 11:21:05 2021] Out of memory: Kill process 28730 (bundle) score 121 or sacrifice child
[Thu Jun  3 11:21:06 2021] Out of memory: Kill process 21392 (bundle) score 120 or sacrifice child
[Thu Jun  3 11:21:07 2021] Out of memory: Kill process 21392 (bundle) score 120 or sacrifice child
[Thu Jun  3 11:21:40 2021] Out of memory: Kill process 29157 (bundle) score 140 or sacrifice child
[Thu Jun  3 11:33:05 2021] Out of memory: Kill process 20162 (bundle) score 117 or sacrifice child
[Thu Jun  3 11:37:39 2021] Out of memory: Kill process 30173 (node) score 119 or sacrifice child
root@momo ~# systemctl --all | grep mastodon
  mastodon-exporter.service                                                       loaded    inactive dead      mastodon-exporter
  mastodon-sidekiq-default.service                                                loaded    inactive dead      mastodon-sidekiq-default
  mastodon-sidekiq-mailers.service                                                loaded    inactive dead      mastodon-sidekiq-mailers
  mastodon-sidekiq-pull.service                                                   loaded    inactive dead      mastodon-sidekiq-pull
  mastodon-sidekiq-push.service                                                   loaded    inactive dead      mastodon-sidekiq-push
  mastodon-sidekiq-scheduler.service                                              loaded    inactive dead      mastodon-sidekiq
  mastodon-streaming.service                                                      loaded    inactive dead      mastodon-streaming
  mastodon-web.service                                                            loaded    active   running   mastodon-web

So I shall test MemoryMax on it and see what happens :)

[ShadowJonathan]

We are also seeing this behaviour

[nolanlawson]

I've long since set up my Mastodon processes to just restart every few hours or so.

There doesn't seem to be any perf/functional penalty for doing this, and it keeps the memory from growing excessively.

[ThisIsMissEm]

Is this still an issue on 4.2.x ?

[untitaker]

I am almost certain it is. I tried to profile memory at some point in 4.0.x and saw that mastodon and rails have a few caches that slowly fill up over time, but no actual leaks. There's two things I have tried, but I cannot recall which ones have made a difference, and restarts have better cost/benefit. I'm on a really small instance so observing memory leaks takes a very long time.

disable query cache

# as config/initializers/querycache-hack.rb
class SidekiqKillArCache
  def call(worker, msg, queue)
    ActiveRecord::Base.uncached do
      yield
    end
  end
end

if ENV['ENABLE_QUERY_CACHE_HACK'] == 'true'
  module Mastodon
    class Application
      config.middleware.delete "ActiveRecord::QueryCache"
    end
  end

  Sidekiq.configure_server do |config|
    config.server_middleware do |chain|
      chain.add SidekiqKillArCache
    end
  end
end

disable jsonld cache

# as config/initializers/jsonld-hack.rb
if ENV['ENABLE_JSON_LD_HACK'] == 'true'
  module JSON::LD
    class Context
      CACHE_SIZE = 0
      @cache = nil
      @inverse_cache = nil
    end
  end
end

[ThisIsMissEm]

I think this is fully expected behaviour? I'm not exactly sure what the caching strategy is, but perhaps it can be memory mapped to use a fixed size and evict old keys? I'm not sure.

[ThisIsMissEm]

@mperham I don't suppose you have any suggestions around Sidekiq and ActiveRecord's caches?

[mperham]

Integrating jemalloc is considered best practice for large multithreaded Ruby apps, I would recommend that if not done already.

Since mastodon has 100s of gem dependencies, it’s impossible to know where a memory leak comes from but native gems are always my first suspects. Upgrade them and follow their changelogs aggressively.

[mperham]

Here’s how to get a list of all native gems: https://github.com/sidekiq/sidekiq/wiki/Problems-and-Troubleshooting#my-sidekiq-process-is-crashing-what-do-i-do

[ThisIsMissEm]

@mperham from what I understand, this is the operating system reaping the process with OOM errors, due to limits imposed on memory consumption; I don't think Ruby is designed to allow this sort of resource constraining?

[mperham]

Correct, there's no way to limit Ruby's memory use "internally". You can only limit the container its in and then restart it once it goes past that limit.

One of the big tech giants (IBM, in my memory) had a patent on their "trick" to getting 100% uptime with their clustered software. The trick? One member of the cluster was always rebooting. Once that instance N is up and running, instance N+1 would reboot. No need to worry about memory or resource leaks if you can guarantee a reboot every N hours.

For this reason, I think it's totally reasonable to restart a puma/sidekiq process every N hours.