Thread leak in start_async

[tikh]

Every time the Slack client is reconnected, a new thread is created. Here is a code to reproduce the issue: https://gist.github.com/tikh/8d43a1a70dca55e89c0645cf7c9f6cc0. Thread count is going up for every reconnect. We are using celluloid-io gem here.

[dblock]

The start_async method is creating a new actor, via Actor.new, we must not be cleaning up the old one correctly, I would look at the code that we run on close, I don't see anything obvious at a first look.

[kandadaboggu]

When you are connecting to few hundred clients the processes runs out of threads very quickly. In last few days the slack server has been disconnecting connection every few minutes. So this has caused servers die often.

[dblock]

Confirming these are Celluloid threads.

threads (num = 6): [
  "#<Thread:0x007fd4508bc410>",
  "#<Celluloid::Thread:0x007fd452ff3510>",
  "#<Celluloid::Thread:0x007fd452fe84d0>",
  "#<Celluloid::Thread:0x007fd452fe0f28>",
  "#<Celluloid::Thread:0x007fd452fd3eb8>",
  "#<Celluloid::Thread:0x007fd452b636a8>"
]

threads (num = 7): [
  "#<Thread:0x007fd4508bc410>",
  "#<Celluloid::Thread:0x007fd452ff3510>",
  "#<Celluloid::Thread:0x007fd452fe84d0>",
  "#<Celluloid::Thread:0x007fd452fe0f28>",
  "#<Celluloid::Thread:0x007fd452fd3eb8>",
  "#<Celluloid::Thread:0x007fd452b636a8>",
  "#<Celluloid::Thread:0x007fd450ae3108>"
]

I'm using

      GC.start
      threads = Thread.list
      objects = ObjectSpace.count_objects
      puts "objcount: #{JSON.pretty_generate(objects)}"
      puts "threads (num = #{threads.count}): #{JSON.pretty_generate(threads)}"

[kandadaboggu]

It looks like we have to call terminate on the old actor.
https://github.com/celluloid/celluloid/wiki/Actor-lifecycle

[dblock]

Yep, just reached the same conclusion, I am with you. Simpler repro:

require 'spec_helper'

RSpec.describe 'reconnect test', skip: (!ENV['SLACK_API_TOKEN'] || !ENV['CONCURRENCY']) && 'missing SLACK_API_TOKEN and/or CONCURRENCY' do
  around do |ex|
    WebMock.allow_net_connect!
    VCR.turned_off { ex.run }
    WebMock.disable_net_connect!
  end

  let(:logger) do
    logger = Logger.new(STDOUT)
    logger.level = Logger::INFO
    logger
  end

  before do
    Thread.abort_on_exception = true

    Slack.configure do |slack|
      slack.logger = logger
    end
  end

  after do
    Slack.config.reset
  end

  let(:client) { Slack::RealTime::Client.new(token: ENV['SLACK_API_TOKEN']) }

  before do
    client.on :hello do
      logger.info "Successfully connected, welcome '#{client.self.name}' to the '#{client.team.name}' team at https://#{client.team.domain}.slack.com."

      @stopping = true
      client.stop!
    end

    client.on :close do
      if @stopping
        @stopping = false
        next
      end

      GC.start
      threads = Thread.list
      objects = ObjectSpace.count_objects
      puts "objcount: #{JSON.pretty_generate(objects)}"
      puts "threads (num = #{threads.count}): #{JSON.pretty_generate(threads)}"

      logger.info 'Closing, and restarting ...'
      sleep 10
      client.start_async
      # TODO: restart
    end
  end

  it 'reconnects in a loop' do
    client.start_async

    loop do
      sleep 1
    end
  end
end

[dblock]

Check out #105. LMK if you think I am missing anything.

[dblock]

Fun graph, before from game-bot: