Logstash 1.4.2/Windows 2012 R2/JDK java version "1.7.0_71" memory leak #2555
Comments
|
If you're seeing a memory leak, the screenshots aren't indicative of it. The first screenshot is a normal-looking sawtooth garbage collection curve. The second is a screenshot of threads with one of the "RMI TCP Connection" threads highlighted - These "RMI TCP Connection" threads only appear when you have attached with VisualVM and are not present otherwise. |
Can you link to those errors?
Can you link to this error? |
|
Well, that was silly of me (JVM/Java newbie). I had stripped everything down to just the Eventlog input and RabbitMQ output, thinking that this RMI TCP connection thing was causing the leak. I appreciate you providing the insight. So the whole truth is that I am using Eventlog and WMI input (from logstash-contrib) along with the RabbitMQ output. Here are the full input files: Eventlog.conf: WMI.conf: On our staging box (win2008) it crashed after running for about 2 hours or so. On my vagrant box it has been about 1 hour and 57 mins and it is still running but maxing out the memory and still growing. Also, notice that VisualVM is reporting a max memory of 256M.
Here is the task manager showing that the JVM process is taking up way more than 256M of memory and almost about to consumer all of it. Time (a): Later Time(b) (and still growing): Later Time (c) (and still growing): Later Time (d) (and still gowing): I am still waiting for it to crash, so as soon as it does, I will upload the errors from the log. |
|
←[31mWMI query error: IEnumVARIANT::Next ←[31mWindows Event Log error: Invoke of: ExecNotificationQuery ["org.racob.com.Dispatch.invokev(Native Method)", "org.racob.com.Dispatch.invoke |
|
BTW, it took 2 hours and 5 mins for it to start erroring and I didn't wait for it to crash because I wanted to get the first error before it could scroll of the screen ... I was running it from the command line. <gut_feeling> Also, I think VisualVM may have been making the situation worse. I am running it now without VisualVM, the Task Manager memory is still growing but it seems not as fast as it was with VisualVM. I will keep you posted on what I find. Any help/advice would be greatly appreciated. |
|
Is this still an issue? |
|
I don't know. My former employer and I parted ways, so I am not longer involved in this effort. I can close this issue. |
Here is to hoping you guys can help with this:
C:\logstash>java -version
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)
C:\logstash>ver
Microsoft Windows [Version 6.3.9600]
JRuby version:
C:\logstash>bin\logstash agent --config conf.d
←[33mUsing milestone 2 input plugin 'eventlog'. This plugin should be stable, bu
t if you see strange behavior, please let us know! For more information on plugi
n milestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:war
n}←[0m
←[33mUsing milestone 1 output plugin 'rabbitmq'. This plugin should work, but wo
uld benefit from use by folks like you. Please let us know if you find bugs or h
ave suggestions on how to improve this plugin. For more information on plugin m
ilestones, see http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}←
[0m
input.Eventlog.conf:
input {
eventlog {
logfile => "System"
tags => [ "system", "windows", "event_log", "system"]
type => "system_windows_event_log_system"
}
}
output.RabbitMq.conf
output {
rabbitmq {
durable => false
exchange => "my-exchange"
exchange_type => "topic"(required)
host => "my-server-name"
key => "some-topic"
password => "my-rabbit-password"
persistent => false
user => "my-rabbit-user"
}
}
And I set the LS_MAX_MEM to be 256m for debugging.
Now the interesting bit is that as I keep this running for about two hours or so, logstash stops publishing and starts filling up the logfile with interesting errors and then eventually the JVM crashes.
Here is a screenshot from VisualVM JVM profiler:
Here is the screenshot highlighting the memory leak:
Referring to the screenshot, please notice the number of threads called "RMI TCP Connection (#number) - {IP_ADDRESS}". This thread list keeps increasing and the "Finished" threads are never garbage collected. Eventually this just crashes the JVM.
Can you please advise on how to further debug the issue? Or suggest how I could resolve the issue.
Thank you.
The text was updated successfully, but these errors were encountered: