Bug: Filehandle Resouce Leak in Colibiri Websocket Proxy #202
Comments
|
The resource leak is proved to be caused by a bug inside the Colibiri Websocket Proxy: Disabling this component (and fall back to HTTP transmission of the Colibiri data channel) on a running system stops further fh consumption for new connection clients, i.e. the overall level of file handles don't rise anymore on a significant way. Here some resource graphs for the Linux Container used for Openfire taken by a monitoring system. There are noticeable growings of the fh's at start of conference meetings (typical at full hour) where a lot of clients start to use the service this day. At about 12:30 there was a manual restart of OpenFire due to the near out-of-filehandle situation. At about 14:30 the use of the Colibiri-WS-Proxy was disabled. As a comparison, here's a snapshot of the next day. The're was a full restart at the evening before and the WS Proxy was kept disabled all the time. |
|
Investigations using VisualVM while having WS Proxy enabled had shown: There are a continouosly rising number of Threads named But the most are 100% at state Running, but seems to futile wait for input at a socket. Example of thread task dump: I guess that this caused by instances of the "innerside" http client connection of between the Colibri WS proxy and the Colibiri Endpoint at the JVB2. |
|
Thanks for all the valuable data and debugging. I found that the proxy was not calling the stop method on the created websocket client object. See 6c4b9f4 Please try latest jar file and confirm if it makes any difference. |
|
Thank you. Will ASAP try to make an A/B-Test first by activating the debug logger |
|
Now, the http clients are alt least closed at the moment a client leave a conference - a big step forward. But each joining client still create 11 http client threads but seems to use just two of it. Below you'll see a part of the live thread panel from VisualVM for the JVM for JVB while two clients are joined at a conference:
Might there something around like a unused, "lost" and default-sized http-client pool? |
|
Below is the output of the |
|
Maybe the Factory The JavaDoc states:
|
|
With help of a good friend, it seems that I was able to trace down the core reason. In the core, one have to replace an automagical pool-sizing mechanism: Because we use a websocket connection over the lifetime of thread, we don't need more than one client. With this change, I observe one http-client thread per incoming WebProxy connection.
We're going to prepare a consolidated pull request. In extension to a fix of the core problem, we suggest to also remove deprecated methods and some refactoring like use a "singleton" SSL context factory. |
|
We still have a -- much smaller -- leak of the triple in some, yet unknown cases. |
With websocket transfer for Colibiri data channel, there's a serious file handle leak. In groups of three fh's, the total number of sockets is increasing all the time.
Invoking a forced FullGC via JMX don't help out, the resources will never be released during uptime of the OpenFire JVM.
The text was updated successfully, but these errors were encountered: