RTMP streams show Extended timestamp exists in HEADER_CONTINUE and server memory fills quickly.

[Mohit-3196]

In one of the customer cases, for RTMP streams we see log entries like:

• Publish RTMP stream on the server continuously 24/7 without stopping the publish.

2023-07-21 14:45:24,884 [NioProcessor-4] INFO  o.r.s.n.r.codec.RTMPProtocolDecoder - Extended timestamp exists in HEADER_CONTINUE last header:Header [streamId=1, channelId=3, dataType=9, timerBase=65038054, timerDelta=0, size=16651, extendedTimestamp=65038054] chunk header:ChunkHeader [type=3, channelId=3, size=1]
2023-07-21 14:45:24,884 [NioProcessor-4] INFO  o.r.s.n.r.codec.RTMPProtocolDecoder - Extended timestamp exists because timevalue is 0xffffff chunkheader:ChunkHeader [type=0, channelId=3, size=1]
2023-07-21 14:45:24,884 [NioProcessor-4] INFO  o.r.s.n.r.codec.RTMPProtocolDecoder - Extended timestamp exists in HEADER_CONTINUE last header:Header [streamId=1, channelId=3, dataType=9, timerBase=65038074, timerDelta=0, size=17088, extendedTimestamp=65038074] chunk header:ChunkHeader [type=3, channelId=3, size=1]

And also there are heap dumps generated on the server.

Logs and heap dumps can be accessed from here https://drive.google.com/drive/folders/1YQYf953o1vvTK6Dsd25pHODXgE7BMO62?usp=sharing

[Mohit-3196]

Also, the antmedia-error.log file size is ever-increasing and it consumes the entire system disk and ultimately fills the space.
The error logs need to be deleted manually to free the disk space.

[mekya]

I'm analyzing the files.

• java_pid14457.hprof: It seems that total memory occupied is 51MB. I think it's a signal that the server configuration and memory size is very low to run Ant Media Server. 37MB of the memory is occupied by ZipFileSource which is zipping the logs.
• java_pid221994.hprof: Total Memory 46.7MB. The same scenario as above.
• java_pid333447.hprof: Total Memory 47.8MB. The same scenario as in java_pid14457
• java_pid336991.hprof: Total Memory 50.5MB. The same scenario as in java_pid14457
• java_pid354843.hprof: Total Memory 47.7MB. The same scenario as in java_pid14457
• java_pid362570.hprof: Total Memory 45.7MB. The same scenario as in java_pid14457

I think for the heapdumps above, there is no problem. It's just about 50MB. If server is giving Out of Memory issue, it means that the virtualization or container has some kind of configuration issues or it's very low profile instance that it gives OOM when heap memory is about 50MB.

• java_pid347586.hprof: Total Memory 818MB. It seems that there is some load on NetSocketImpl. I'm digging into this.

One instance of io.vertx.core.net.impl.NetSocketImpl loaded by org.red5.server.classloading.ServerClassLoader @ 0x89388fa0 occupies 807.546.704 (94,14%) bytes. The memory is accumulated in one instance of byte[], loaded by , which occupies 807.545.912 (94,14%) bytes.

[mekya]

The problem is about that some other client connects to 5000 port(which should be closed to public) and send other messages that mess up the server. We see this as opportunity to increase security between node communication

[mekya]

Closing this issue because fix for this implementation has been merged to the master. Please feel free to re-open if it does not work for you