Actors/Threads remain after calling SupervisionGroup#finalize and terminate #203

martyMM · 2013-03-29T15:49:51Z

Running into a problem where threads are remaining after their SupervisionGroup has been terminated. You can reproduce it by grabbing the script below, launching a console and requiring the file. See my output below. Also notice the error when calling Celluloid.stack_dump immediately after completion. However, waiting a few minutes will let stack_dump run successfully.

Tested on JRuby 1.7.2

Script: https://gist.github.com/martyMM/5271674
open IRB and require the file

Total Actors: 7
Working...
Working...
Working...
Working...
Working...
Working...
Working...
Working...
Working...
Working...
Finalizing...
Total Actors: 15
=> true
irb(main):002:0> Celluloid.stack_dump
Celluloid::Actor 0x81a: DomainWorker
State: Idle (waiting for messages)
    NoMethodError: undefined method `backtrace' for nil:NilClass
    from org/jruby/RubyBasicObject.java:1659:in `__send__'
    from /Users/admin/.rvm/gems/jruby-1.7.2/gems/celluloid-0.12.4/lib/celluloid/stack_dumper.rb:42:in `display_backtrace'
    from /Users/admin/.rvm/gems/jruby-1.7.2/gems/celluloid-0.12.4/lib/celluloid/stack_dumper.rb:32:in `dump'
    from org/jruby/RubyHash.java:1257:in `each'
    from /Users/admin/.rvm/gems/jruby-1.7.2/gems/celluloid-0.12.4/lib/celluloid/stack_dumper.rb:15:in `dump'
    from /Users/admin/.rvm/gems/jruby-1.7.2/gems/celluloid-0.12.4/lib/celluloid.rb:45:in `stack_dump'
    from (irb):2:in `evaluate'
    from org/jruby/RubyKernel.java:1066:in `eval'
    from org/jruby/RubyKernel.java:1392:in `loop'
    from org/jruby/RubyKernel.java:1174:in `catch'
    from org/jruby/RubyKernel.java:1174:in `catch'
    from /Users/admin/.rvm/rubies/jruby-1.7.2/bin/irb:13:in `(root)'

What is also interesting is that when watching with VisualVM, all of the JRubyWorker threads exit after a while (presumably these are the pool workers that were launched), but a lot of RubyThreads remain until I kill the console process.

The text was updated successfully, but these errors were encountered:

martyMM · 2013-03-29T15:59:21Z

To add some more context, my actual app dynamically generates supervisor names to focus workers on a specific queue. So I run 100 different jobs, there are 100 different supervisor keys generated. Their worker pools get shut down, but those internal "RubyThread" never exit and the result is a lot of threads continue to run until I run out of memory. This happened on Heroku as well.

According to VisualVM, the "RubyThread"s are coming from /Users/admin/.rvm/gems/jruby-1.7.2/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:45.

tarcieri · 2013-03-29T16:48:00Z

On Fri, Mar 29, 2013 at 8:59 AM, Marty McKenna notifications@github.comwrote:

To add some more context, my actual app dynamically generates supervisor
names

I'm guessing your problem lies here somewhere...

Tony Arcieri

martyMM · 2013-03-29T16:51:52Z

It happens locally with 1 supervisor as well. Those "RubyThreads" never go away. If I launch 2 supervisors, its the same deal. Is that expected behavior?

edit: I mentioned my app on Heroku because you would never notice a problem if it was just one supervisor, because the same threads get reused. But I would assume the threads should still exit when the supervisor is finalized or terminated, right?

tarcieri · 2013-03-29T16:56:38Z

Celluloid uses a thread pool, and the threads of terminated actors remain
inside of it even after the actors are terminated. That's likely the source
of your confusion.

On Fri, Mar 29, 2013 at 9:51 AM, Marty McKenna notifications@github.comwrote:

It happens locally with 1 supervisor as well. Those "RubyThreads" never go
away. If I launch 2 supervisors, its the same deal. Is that expected
behavior?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/203#issuecomment-15649662
.

Tony Arcieri

martyMM · 2013-03-29T17:47:27Z

Is there a way to kill all of the threads and/or thread pool spawned by the supervisor when terminating it?

martyMM · 2013-03-29T20:17:17Z

I've updated my example (https://gist.github.com/martyMM/5271674). If I start a regular pool without a supervisor, the thread counts add up when the pool is terminated. If I start a pool, terminate it, then start an actor, the thread counts also add up.

The thread counts do not add up when creating a pool with a supervisor. Also, when running the try_pool_and_actor multiple times, the thread counts zero out at first, but eventually become +1 as well as increasing by 1 for each call and never go down: https://gist.github.com/martyMM/5273334

tarcieri · 2013-03-29T21:05:40Z

Here's Celluloid's internal thread pool:

https://github.com/celluloid/celluloid/blob/master/lib/celluloid/internal_pool.rb

You can disable it with:

Celluloid.internal_pool.max_idle = 0

On Fri, Mar 29, 2013 at 1:17 PM, Marty McKenna notifications@github.comwrote:

I've updated my example (https://gist.github.com/martyMM/5271674). If I
start a regular pool without a supervisor, the thread counts add up when
the pool is terminated. If I start a pool, terminate it, then start an
actor, the thread counts also add up.

The thread counts do not add up when creating a pool with a supervisor.
Also, when running the try_pool_and_actor multiple times, the thread counts
zero out but increase by 1 for each call and never go down:
https://gist.github.com/martyMM/5273334

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/203#issuecomment-15658683
.

Tony Arcieri

tarcieri · 2013-03-29T21:06:36Z

What you should confirm... either:

Unbounded thread growth with the pool on

or -

Threads persist with the pool disabled

On Fri, Mar 29, 2013 at 2:05 PM, Tony Arcieri tony.arcieri@gmail.comwrote:

Here's Celluloid's internal thread pool:

https://github.com/celluloid/celluloid/blob/master/lib/celluloid/internal_pool.rb

You can disable it with:
Celluloid.internal_pool.max_idle = 0
On Fri, Mar 29, 2013 at 1:17 PM, Marty McKenna notifications@github.comwrote:

I've updated my example (https://gist.github.com/martyMM/5271674). If I
start a regular pool without a supervisor, the thread counts add up when
the pool is terminated. If I start a pool, terminate it, then start an
actor, the thread counts also add up.

The thread counts do not add up when creating a pool with a supervisor.
Also, when running the try_pool_and_actor multiple times, the thread counts
zero out but increase by 1 for each call and never go down:
https://gist.github.com/martyMM/5273334

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/203#issuecomment-15658683
.

Tony Arcieri

Tony Arcieri

martyMM · 2013-04-03T14:27:37Z

This particular issue was with 2) Threads persist with the pool disabled

tarcieri · 2013-04-03T16:03:47Z

Can you post a Celluloid.stack_dump after you've terminated the allegedly lingering actors?

halorgium · 2013-04-03T21:30:18Z

@martyMM could you please also update the gist with a reproduction which produces these leaked Threads directly?
I note you don't actually execute any code just define some methods.

halorgium · 2013-04-12T15:47:33Z

I am able to reproduce this issue on jruby 1.7.2 and 1.7.4-dev.
For reference:

$ irb -r ./sticky_threads
> Celluloid.internal_pool.max_idle = 0; mgr = DomainManager.new; 10.times { mgr.try_pool_and_actor }

jruby

> Thread.list.count
=> 10

MRI

Thread.list.count
=> 1

halorgium · 2013-04-13T01:17:36Z

One interesting thing to note is that the Threads eventually dissappear.
Some related issues of note:
http://jira.codehaus.org/browse/JRUBY-4952
http://jira.codehaus.org/browse/JRUBY-7081

Also, there is an issue with finalizers where they can be terminated prematurely.

halorgium · 2013-04-13T05:11:18Z

I think this is a point of confusion rather than being a bug with Celluloid.
As is documented on the Gotchas wiki page, this is a @jruby errata.

tarcieri · 2013-04-14T20:15:01Z

Fiber thread pool would definitely be another explanation

halorgium · 2013-04-14T20:52:37Z

That is the issue here. I reopened your bug with @jruby...

halorgium mentioned this issue Apr 13, 2013

Finalizers which suspend are terminated immediately #212

Closed

halorgium closed this as completed Apr 13, 2013

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Actors/Threads remain after calling SupervisionGroup#finalize and terminate #203

Actors/Threads remain after calling SupervisionGroup#finalize and terminate #203

martyMM commented Mar 29, 2013

martyMM commented Mar 29, 2013

tarcieri commented Mar 29, 2013

martyMM commented Mar 29, 2013

tarcieri commented Mar 29, 2013

martyMM commented Mar 29, 2013

martyMM commented Mar 29, 2013

tarcieri commented Mar 29, 2013

tarcieri commented Mar 29, 2013

martyMM commented Apr 3, 2013

tarcieri commented Apr 3, 2013

halorgium commented Apr 3, 2013

halorgium commented Apr 12, 2013

halorgium commented Apr 13, 2013

halorgium commented Apr 13, 2013

tarcieri commented Apr 14, 2013

halorgium commented Apr 14, 2013

Actors/Threads remain after calling SupervisionGroup#finalize and terminate #203

Actors/Threads remain after calling SupervisionGroup#finalize and terminate #203

Comments

martyMM commented Mar 29, 2013

martyMM commented Mar 29, 2013

tarcieri commented Mar 29, 2013

martyMM commented Mar 29, 2013

tarcieri commented Mar 29, 2013

martyMM commented Mar 29, 2013

martyMM commented Mar 29, 2013

tarcieri commented Mar 29, 2013

tarcieri commented Mar 29, 2013

martyMM commented Apr 3, 2013

tarcieri commented Apr 3, 2013

halorgium commented Apr 3, 2013

halorgium commented Apr 12, 2013

halorgium commented Apr 13, 2013

halorgium commented Apr 13, 2013

tarcieri commented Apr 14, 2013

halorgium commented Apr 14, 2013