Possible Memory leak

[dohst]

Hello,
i tested a long running Web Conference (>1h) with OpenMeetings 3.0.7 (current version). After a while a OutOfMemopryException occured during screen recording. After getting the heap dump it seems that there is a leak in red5-client (1.0.6-RELEASE) or in the used Apache MINA. I attached a Screenshot from Eclipse MAT

Can you have a look please? Or is the leak in the meantime fixed?
Many thanks and kind regards,
Dirk

[solomax]

Actually I can see weird behavior with our screen-sharing application based on red5-client: it seems packets are not being sent (I do push video tags every 100ms but get weird, laggy picture)
Same time recording seems to look OK

@mondain can it be caused by some delay/caching in mina?

[mondain]

@solomax do you have something i can run a test with? I know the Mina team patched some filter chain stuff recently, but none of our builds have the .10+ release yet.

[solomax]

I can try to provide you with some "debuggable" version of our screen-sharing applet
please let me know if it is OK

[mondain]

Yeah send that over, its more helpful than having to write something up myself.

[solomax]

@dohst Can you please check if the issue is still reproducible using latest build from here: https://builds.apache.org/view/M-R/view/OpenMeetings/job/Openmeetings%203.1.x/
If yes, could you please provide more detailed configuration to reproduce the issue

not reproducible here using

1. local OM 3.1.0 server
2. Windows 7, Ubuntu as a client
   1 hour+ recording works as expected

[dohst]

@solomax
I just downloaded the latest SNAPSHOT and ran into two problems.
I have solved the first: in the *.jnlp file was localhost:5080 aus url
But I cannot solve the second 👍
I get a NetConnection.Connect.Failed in the client and an exception on the server side:

DEBUG 01-30 14:45:24.787 MeetingReminderJob.java 214474 35 org.apache.openmeetings.service.quartz.scheduler.MeetingReminderJob [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8] - MeetingReminderJob.execute
[INFO] [RTMPConnectionExecutor-3] org.apache.openmeetings.core.remote.red5.ScopeApplicationAdapter - W3C x-category:session x-event:connect c-ip:89.0.128.100 c-client-id:1
DEBUG 01-30 14:45:33.263 ScopeApplicationAdapter.java 222950 151 org.apache.openmeetings.core.remote.red5.ScopeApplicationAdapter [RTMPConnectionExecutor-3] - roomConnect :
DEBUG 01-30 14:45:33.263 ScopeApplicationAdapter.java 222950 156 org.apache.openmeetings.core.remote.red5.ScopeApplicationAdapter [RTMPConnectionExecutor-3] - ### Client connected to OpenMeetings, register Client StreamId: 1 scope 9
DEBUG 01-30 14:45:33.265 ManageCryptStyle.java 222952 32 org.apache.openmeetings.util.crypt.ManageCryptStyle [RTMPConnectionExecutor-3] - getInstanceOfCrypt:: configKeyCryptClassName: org.apache.openmeetings.util.crypt.MD5Implementation
[ERROR] [RTMPConnectionExecutor-3] org.red5.server.net.rtmp.RTMPHandler - Error connecting {}
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at org.apache.openmeetings.core.remote.red5.ScopeApplicationAdapter.roomConnect(ScopeApplicationAdapter.java:189) ~[openmeetings-core-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
at org.red5.server.adapter.MultiThreadedApplicationAdapter.connect(MultiThreadedApplicationAdapter.java:315) ~[red5-server.jar:1.0.6-RELEASE]
at org.red5.server.adapter.ApplicationAdapter.connect(ApplicationAdapter.java:42) ~[red5-server.jar:1.0.6-RELEASE]
at org.red5.server.scope.Scope.connect(Scope.java:255) ~[red5-server-common-1.0.6-RELEASE.jar:na]
at org.red5.server.BaseConnection.connect(BaseConnection.java:354) ~[red5-server-common-1.0.6-RELEASE.jar:na]
at org.red5.server.net.rtmp.RTMPConnection.connect(RTMPConnection.java:405) ~[red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.RTMPMinaConnection.connect(RTMPMinaConnection.java:87) ~[red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.RTMPHandler.onCommand(RTMPHandler.java:347) ~[red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:105) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:63) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at org.red5.server.net.rtmp.ReceivedMessageTask.call(ReceivedMessageTask.java:14) [red5-server-common-1.0.6-RELEASE.jar:1.0.6-RELEASE]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_72]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
[INFO] [RTMPConnectionExecutor-3] org.apache.openmeetings.core.remote.red5.ScopeApplicationAdapter - W3C x-category:session x-event:disconnect c-ip:89.0.128.100 c-client-id:1
DEBUG 01-30 14:45:45.311 MeetingReminderJob.java 234998 35 org.apache.openmeetings.service.quartz.scheduler.MeetingReminderJob [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-8] - MeetingReminderJob.execute
TRACE 01-30 14:45:51.614 ScopeApplicationAdapter.java 241301 1728 org.apache.openmeetings.core.remote.red5.ScopeApplicationAdapter [Thread-12] - [MessageSender] -> 'sending message' sendVarsToMessage, [typingActivity, true, 1235ce2f33df485ded0f615d2f07a40d]

I attached the screenshare log.

screensharing.log.txt

[mondain]

Ill take a look after i get some other items resolved server-common.

[solomax]

@dohst I'll double-check this ASAP

[solomax]

@dohst build #131 should work as expected
@mondain the following code will change Long to Integer on server:
RTMPClient instance = new RTMPClient();
Map<String, Object> map = instance.makeDefaultConnectionParams(host, port, app);
map.put("userId", (Long)userId); //Long here
instance.connect(host, port, map, this);
On server map.get("userId") will be Integer, not Long

[mondain]

The problem we'll run into there is that with AMF0, there is nothing but Double or as Adobe says: Number; so internally it will convert 1 to 1.0 for storage.

[solomax]

And how Long become Integer in this case?

[mondain]

That I cannot explain, reading through the AMF0 spec, it says all numbers are Double; only AMF3 allows Integer. There are however unsigned shorts and ints used for Array and String sizes.

[dohst]

Uploading screensharing.log.txt…

Today I tried it wirh the mentioned build on Client: Win 7 Pro (8GB RAM) and Server: Debian 8. After about 50 minutes the OoM occured again. see attached log file
In the meantime we switched to WebEx because it is cheaper for 70+ participiants than a dedicated server with enough CPU power to host OM.
Many thanks for your support and good luck and joy.