Concurrent access, memory leak

[myles1231]

The version number of the SRS being tested is 2.0.239.
Configuration File
Source configuration file origin.conf

listen              19350;
max_connections     1000;
daemon              off;
#srs_log_level       verbose;
srs_log_tank        console;
pid                 ./objs/origin.pid;
vhost __defaultVhost__ {
}

Edge 1 configuration file edge1.conf

listen              1935;
max_connections     1000;
pid                 objs/edge1.pid;
daemon              off;
srs_log_level       info;
#verbose;
srs_log_tank        console;
vhost __defaultVhost__ {
    mode            remote;
    origin          127.0.0.1:19350;
}

Edge 2 configuration file

listen              19358;
max_connections     1000;
pid                 objs/edge2.pid;
daemon              off;
srs_log_level       info;
#verbose;
srs_log_tank        console;
vhost __defaultVhost__ {
    mode            remote;
    origin          127.0.0.1:19350;
}

Test Steps

1. Edge 1 only pushes the stream, and Edge 2 only pulls the stream. To confirm whether it is a push leak or a pull leak, separate the push and pull streams.
2. Push different streams to Edge 1. The script for pushing on Ubuntu is as follows:

#!/bin/bash
i=

for((;;)); do \
    ffmpeg -re -i 200kbps.768x320.100s.flv \
        -vcodec copy -acodec copy \
        -f flv -y rtmp://192.168.149.128/live/livestream$i; \
    sleep 1; \
    i=`expr $i + 1`
done

Test Steps
3. Continuously pull changing streams from Edge 2 and the source end.

Results
The SRS version number is 2.0.239. When cleaning and closing the SrsSource, it was found that Edge 2's memory was increasing much faster than the source end. Later, after enabling the cleaning of SrsSource, it was observed that the memory of the source end and Edge 1 stopped increasing, while Edge 2's memory increased by 1M every half an hour. The memory did not decrease after stopping the stream. It remained constant.

It was confirmed that the pool size of Edge 2's SrsSource did not keep increasing and was eventually cleaned. Additionally, even after adding 803 to STATISTIC: Cleanup stream when closed #803, the memory leak issue remained the same. It seems that there might be other memory leaks in the stream pulling part of the edge.

Currently, there are no clear clues, so if anyone can help take a look, it would be greatly appreciated. Thank you~

TRANS_BY_GPT3

[myles1231]

It doesn't matter if the discovered flow name is different, I only push one flow and start three processes to pull the flow. Later, using a tool to pull 1000 concurrent flows, the generated memory is not released even after stopping the flow. It should be a leak caused by concurrency.

TRANS_BY_GPT3

[winlinvip]

Must use tools to find: http://blog.csdn.net/win_lin/article/details/53503869
Concurrent testing with SB: https://github.com/ossrs/srs-bench

TRANS_BY_GPT3

[winlinvip]

Note that SRS2 does not support VALGRIND, only SRS3 does. You can start by looking for it in SRS3.

TRANS_BY_GPT3

[myles1231]

OK! Currently using SRS3, VALGRIND calls.

TRANS_BY_GPT3

[myles1231]

Currently, no free memory has been found, but even after freeing it, the system does not reclaim it. Instead, the memory keeps increasing, which could be due to a memory leak in glibc's ptmalloc.

TRANS_BY_GPT3

[myles1231]

The following information appears on the playback end, but the streaming end does not.
srs memory watch leak report:
RTMP.msg.payload: 0x267b880, 4B
RTMP.msg.payload: 0x2692690, 4B
RTMP.msg.payload: 0x2695e20, 46B
RTMP.msg.payload: 0x26afdf0, 46B
RTMP.msg.payload: 0x26b75b0, 4B
RTMP.msg.payload: 0x26b8d40, 46B
RTMP.msg.payload: 0x26ca110, 4B
RTMP.msg.payload: 0x26e2a90, 46B
RTMP.msg.payload: 0x2732130, 4B
RTMP.msg.payload: 0x2774940, 46B
10 objects leak 0KB.
@remark use script to cleanup for memory watch: ./etc/init.d/srs stop

TRANS_BY_GPT3

[winlinvip]

The player is a cache for gop cache.
Just because the memory increases doesn't necessarily mean there is a leak, unless it keeps increasing without stopping.

TRANS_BY_GPT3

[myles1231]

I have already cleaned up SrsSource and cleared the gop cache. The memory has indeed been increasing continuously.

TRANS_BY_GPT3

[rukesun]

I use v2 version, and this issue is easy to reproduce.

Test steps:

1. Push a stream to the origin node.
2. Use srs bench to concurrently pull the stream from the edge node with 300 connections (note that only the same stream is pulled).
3. Repeat step 2 several times (I tested 5 times) and observe that the memory usage of the edge node keeps increasing (note that the memory increases by a few megabytes after each test, and it does not decrease after stopping the test. In other words, it is related to the total number of connections established in history. However, the SrsRtmpConn::~SrsRtmpConn destructor method is called by using breakpoints).

Using the pprof tool to inspect:
gperf/bin/pprof ./srs "/tmp/srs.29507.main-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10

Note: I have a total of 1500 RTMP connections, suspecting it is related to memory allocation in ST (when is _st_stack_free called?), _st_stack_new about 1500 times. Each time there are 300 concurrent connections, _st_netfd_new about 300 times. Below are the results after stopping the stress test.

(pprof) top
Total: 3626 objects
1584 43.7% 43.7% 1584 43.7% __gnu_cxx::new_allocator::allocate /home/shared/deps/include/c++/5.2.0/ext/new_allocator.h:105
1507 41.6% 85.2% 1507 41.6% _st_stack_new /xxx/trunk/objs/st-1.9/stk.c:78
305 8.4% 93.7% 305 8.4% _st_netfd_new /xxx/trunk/objs/st-1.9/io.c:144
58 1.6% 95.3% 58 1.6% SrsConfDirective::parse_conf /xxx/trunk/src/app/srs_app_config.cpp:297
34 0.9% 96.2% 34 0.9% _M_dispose (inline) /home/shared/pkgs/gcc-5.2.0/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:135
18 0.5% 96.7% 18 0.5% nss_parse_service_list nsswitch.c:0
16 0.4% 97.1% 16 0.4% __nss_database_lookup ??:0
12 0.3% 97.5% 12 0.3% std::__cxx11::basic_string::_M_construct /home/shared/pkgs/gcc-5.2.0/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:223
5 0.1% 97.6% 10 0.3% __static_initialization_and_destruction_0@53fd78 srs_main_server.cpp:0
4 0.1% 97.7% 4 0.1% SrsPithyPrint::enter_stage /xxx/trunk/src/app/srs_app_pithy_print.cpp:173

TRANS_BY_GPT3

[rukesun]

Ignore me, caused by compiling with the --with-gperf --with-gmc options.

TRANS_BY_GPT3

[winlinvip]

Fixed by SmartPtr: Support shared ptr for live source. v6.0.129