Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to stop Netty 4 Timer #757

Closed
dani-e-l opened this issue Feb 6, 2019 · 6 comments
Closed

Unable to stop Netty 4 Timer #757

dani-e-l opened this issue Feb 6, 2019 · 6 comments

Comments

@dani-e-l
Copy link
Contributor

dani-e-l commented Feb 6, 2019

Hello we are using Finagle as a HttpClient - it works really good. Great job.

Unfortunately Finagle generates resource leak under standard behavior that applications are deployed and undeployed many times on JBoss AS, Tomcat etc.
By resource leak I mean that if I 100 times deploy and undeploy an application that uses Finagle I'll get at least 100 threads

The problem is that finagle contains unstopable threads

eg.

private[netty4] class Netty4HashedWheelTimer
    extends Netty4Timer(HashedWheelTimer.instance)
    with ServiceLoadedTimer {

  private[this] val log = Logger.get()

  // This timer is "unstoppable".
  override def stop(): Unit =
    log.warning(
      s"Ignoring call to `Timer.stop()` on an unstoppable Netty4Timer.\n" +
        s"Current stack trace: ${Thread.currentThread.getStackTrace.mkString("\n")}"
    )
}
@kevinoliver
Copy link
Contributor

Ah. I suspect there may be other issues like this as haven't designed for that type of deployment model.

This might require a patch to create a flag that allows you to opt-out of this behavior.

@dani-e-l
Copy link
Contributor Author

dani-e-l commented Feb 7, 2019

Thanks for quick reply Kevin.
I was able to close many other threads because API allows me to use my own Timers/Executors/etc
I have problem with last three threads.

In this case private object is used and I have no access to it.

@mpnewcomb
Copy link

HashedWheelTimer... why are you using this? It is a cpu hog. Start up linkerd and namerd and they just sit their chewing anywhere from 1-3% cpu when nothing is happening...

@kevinoliver
Copy link
Contributor

@mpnewcomb do you have any suggestions on how to improve it? if there is no work to do, it should be a thread that sleeps for a few milliseconds, looks for work, then repeats.

@drywet
Copy link

drywet commented May 29, 2019

Hello. This seems to work for me:
Create an object with a method in package com.twitter.finagle.netty4 to access private properties and stop com.twitter.finagle.util.HashedWheelTimer.nettyHwt
and
HashedWheelTimer.instance
timers

dani-e-l pushed a commit to dani-e-l/finagle that referenced this issue Oct 4, 2019
twitter-service pushed a commit that referenced this issue Oct 9, 2019
Problem

No possibility to stop "Netty 4 Timer".
Detailed description is available here: [[
#757 | #757 ]]

Solution

Make HashedWheelTimer object public with additional method to stop "Netty 4
Timer"

Result

Possibility to stop "Netty 4 Timer"

Signed-off-by: Jing Yan <jyan@twitter.com>
GitOrigin-RevId: 290708fd02090af60d64462c8da5a0d5a2b94f99
finaglehelper pushed a commit that referenced this issue Oct 9, 2019
Problem

No possibility to stop "Netty 4 Timer".
Detailed description is available here: [[
#757 | #757 ]]

Solution

Make HashedWheelTimer object public with additional method to stop "Netty 4
Timer"

Result

Possibility to stop "Netty 4 Timer"

Signed-off-by: Jing Yan <jyan@twitter.com>

Differential Revision: https://phabricator.twitter.biz/D381605
@tigerlily-he
Copy link
Contributor

This feature to stop a Netty 4 Timer was added in fa4899e. Closing this issue now that it's resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

5 participants