Memory leak in 5.0.1

[key88sf]

We are using a Datadog::Statsd object in a sidekiq worker. When the worker executes, we basically do this:

statsd = Datadog::Statsd.new("localhost", 8125)
statsd.increment(....) # specific params not included here

When upgrading from 4.8.3 to 5.0.1, we are seeing memory usage on the instance start to climb linearly until it finally exhausts all memory and we get a ThreadError: can't create Thread: Resource temporarily unavailable. We have definitively pinpointed this problem to the 5.0.1 upgrade --- there were no other changes made other than upgrading just the dogstatsd-ruby gem.

You can see the mem usage problem in the graph below:

(each little dip is a deploy where we changed just one gem version. The last one is where we upgraded dogstatsd-ruby from 4.8.3 to 5.0.1).

[remeh]

Hey @key88sf
Thank you for having reported this issue.

I suspect this may be because of some resources opened in every client that your worker is creating, since the client is never closed in your snippet (#181 will add an auto-close feature on the client in 5.0.2).

Meanwhile, in your code, would it be possible to close the opened client? E.g.

statsd = Datadog::Statsd.new("localhost", 8125)
statsd.increment(....) # specific params not included here
# ... and near the end of the code execution ...
statsd.close()

Otherwise, we'll soon release 5.0.2.

[key88sf]

@remeh I will try this and let you know soon. Thanks!

UPDATE: Actually going to wait until 5.0.2. Sorry! Priorities.... :(

[ArmandoAssuncao]

I have a same issue, in version 5.x, the sender.rb:72 create a lot of threads:

versions:

• gem 'ddtrace', '~> 0.48.0'
• gem 'dogstatsd-ruby', '~> 5.0.1'

I currently downgrade version to 4.8.3 while wait the new version.

[remeh]

Hey @ArmandoAssuncao, thanks for your comment.
Are you using DogStatsD-ruby in a Sidekiq worker manually or are you using something else? Since version v5.x, a thread is created when a new DogStatsD client is instantiated (and is closed when the client is closed). In your situation where we can indeed see multiple threads being created, I think that multiple clients are created and are never closed.
That's why if you manage the lifecycle of your DogStatsD clients, please ensure that they're closed when not needed anymore.

[ArmandoAssuncao]

@remeh I use gem ddtrace that uses the DogStatsD-ruby internally.

Only config that has is:

# config/initializers/datadog.rb
if Rails.env.production?
  Datadog.configure do |c|
    c.use :sidekiq, service_name: "name_sidekiq", client_service_name: "name_sidekiq", tag_args: true
  end
end

Probably the ddtrace not call method close correctly.

[marcotc]

👋, I've created a cross issue at ddtrace to track this report, as it seem to related to dogstastd-ruby usage by ddtrace: DataDog/dd-trace-rb#1491

[deepfryed]

@marcotc ran into the same issue, seeing unbounded growth with 5.0.1 and dd-trace-rb disabled (skipping the config.use :rails, ...). Seeing a slow growth in sender.rb threads, which seems to be the source of the memory leak.

downgrading to 4.8.3 results in things returning to normal

We've disabled APM for the time being and downgraded dogstatsd-ruby until a fix is out.

[marcotc]

Thank you for the information, @deepfryed. Are you also using Sidekiq, and had c.use :sidekiq in your Datadog.configure do |c|... block for your ddtrace configuration?

[deepfryed]

@marcotc not using sidekiq in the rails app, ddtrace is required but not used as noted in my comment.

[marcotc]

I've tried to reproduce this issue with both a Sidekiq and a Rails application, but with no success: both versions have the same memory profile regardless of dogstatsd-ruby being version 4.8.3 or 5.0.1.

👋 @ArmandoAssuncao, given you have a clear output pointing to sender.rb:72 being the culprit, I've created a Sidekiq test run that runs the Sidekiq server and schedules 10000 jobs, but I don't see unbounded memory accumulation, nor additional threads being created.

This is likely due to a difference between what my Sidekiq server is doing, compared to yours.

Could I ask you to provide us with your Sidekiq configuration:

• Version of Sidekiq and Redis gem.
• sidekiq.yml
• The command executed to run Sidekiq.
• Additional "runner" information, if applicable. (e.g does it run under puma? If so, what's your puma configuration?).

Additionally, could you provide a sample Sidekiq worker class? For the worker class, let me know what kind of pattern you normally use to schedule them (e.g. delayed job, perform_after). I'd like to trigger a similar pattern myself to try to reproduce the issue.

Regarding Datadog configuration, I noticed you provided your Datadog.configure do |c|... block above: is that the complete block? If not, could you provide the complete block instead? Feel free to mask any private information (e.g. services names).

Still regarding Datadog.configure, do you have any special configuration to make it work with the Sidekiq worker process? I ask because sometimes, depending on your setup (e.g. non-Rails), you might have to manually initialize Datadog.configure for Sidekiq jobs to be instrumented.

Sorry for asking so much information, but I've inspected both ddtrace and dogstatsd-ruby for clues and it doesn't seem like it's possible for ddtrace to ever create more than 2 dogstatsd-ruby at any point in time. There's also no dynamic dogstatsd-ruby instance creation: it only happens during Datadog.configure calls. And days of testing with the application types suggested here yielded no reproduction.

My current priority is to fix this issue, so let me know if you can't provide the information above for any reason.

Thank you in advance!

[deepfryed]

@marcotc

this is what we got when checking our production container running dogstatsd 5.0.1. No sidekiq, the puma (5.2.2) config has max 32 threads.

root@app-prod-6949448b68-vj4q6:/app# ps -eL | grep sender.rb | wc -l
1091

The rails (6.1) config is below

Datadog.configure do |c|
  # This will activate auto-instrumentation for Rails when not in test environment 
  # and enable log traceid injection
  c.use :rails, log_injection: true, database_service: 'server-postgres', cache_service: 'server-redis'
end

Datadog::Pipeline.before_flush(
  # Remove spans for /up requests
  Datadog::Pipeline::SpanFilter.new { |span| span.resource =~ /InfoController/ }
)

[marcotc]

Thank you, @deepfryed! This is helpful, I'm thinking puma might be an important component on this that I haven't tested with.

Are there any other references to Datadog gems (either ddtrace or dogstatsd-ruby) in your application's code?
Also, what version of ddtrace are you running?