High memory usage

[drobotk]

Hi, I've noticed some very high memory usage by this library, possibly a memory leak.

Whenever I stream anything to my receiver peer, my RAM usage linearly goes up until it overflows and my PC slows down to the point of having to reboot. The rate at which the usage increases seems to depend on the input file I decide to stream. I've also noticed that (strangely enough) the memory usage seems higher when sending only audio and no video.

I have also tried to add an event listener for receiving remote tracks, in case I need to discard them using the MediaBlackhole, but it never gets called, so no tracks to handle.

Codecs: opus & H264

I don't know what other data might be relevant for troubleshooting this issue, so please comment in case I should send anything.

Thanks in advance

[jlaine]

Can you reproduce this behaviour with an unmodified example, if so which one? I really have nothing to go on here.

[drobotk]

I've fired up the webcam example and Yes, I'm getting the same behaviour. Memory usage linearly goes up until the track ends or I stop the streaming manually - then the usage instantly drops back down.

[jlaine]

Ok, so let's focus on the webcam example if that one triggers the issue reproducibly. Are you running the latest version of PyAV as a memory leak was fixed in 9.2.0?

[drobotk]

I've been using 9.1.0 when I noticed the issue. I've just updated it to 9.2.0, tested again and the issue still occurs.

[drobotk]

Any updates on this? I would really like to continue using this library but this issue is slowing down the development of my project.

I'm really open for collaboration. Is there any other info I can provide? I would be more than happy if I could help resolve this problem.

[jlaine]

No updates to report here, this is not a free help hotline! Profiler traces showing where the memory allocation is happening would help.

Also a fairly obvious question: is video playback falling behind? If so that would indicate you are not able to encode the video in real-time, and I would indeed expect memory usage to grow as the the backlog builds up.

[dfybc]

I have met the same problem when i run the webcam example on raspberry 4B(ram 4G),
the memory usage increases until memory is consumed!!

[W3AXL]

My memory issues from #680 have not gone away entirely either. After a day or two of streaming audio, I get to the max memory for my VM (8 GB) and my script crashes.

Interestingly none of the memory traces I've ran have shown the memory being consumed. The standard tracemalloc output shows the highest memory user as only consuming a few MB of RAM, so something is eating up all the available ram on the server and not being obvious about it.

[W3AXL]

Ran some memory profiling outside of python itself using jemalloc to try and nail down where all the memory usage is coming from, since as mentioned above the internal python memory profilers aren't able to see the memory consumption.

Here's an example heap trace and graph from 30 minutes of running a bidirectional audio stream. This is the same script that balloons to > 4GB over a few days, but with profiling running I can only run it for 30 minutes to an hour before all my swap is used up.

jeprof --show_bytes `which python` jeprof.35153.0.f.heap
Using local file /home/w3axl/python-radio-console-dev/bin/python.
Using local file jeprof.35153.0.f.heap.
Welcome to jeprof!  For help, type 'help'.
(jeprof) top
Total: 49744145 B
45583104  91.6%  91.6% 45583104  91.6% av_malloc@@LIBAVUTIL_56
 3497297   7.0%  98.7%  3497297   7.0% prof_backtrace_impl
  375552   0.8%  99.4%   729704   1.5% _dl_rtld_di_serinfo@@GLIBC_PRIVATE
   95360   0.2%  99.6%    95360   0.2% asn1_perror@@LIBTASN1_0_3
   71104   0.1%  99.8%    71104   0.1% _dl_debug_state@@GLIBC_PRIVATE
   65600   0.1%  99.9%    66864   0.1% snd_shm_area_destructor@@ALSA_0.9
   12840   0.0%  99.9%    12840   0.0% g_malloc0
   12800   0.0%  99.9%    12800   0.0% pixman_image_get_format
    4464   0.0%  99.9%     4464   0.0% srtp_crypto_alloc
    4128   0.0% 100.0%     5504   0.0% _gfortran_set_max_subrecord_length@@GFORTRAN_8@1c2c0

So it appears the offender is somewhere inside libavutil, which I'm guessing gets used by PyAV/ffmpeg during audio processing. This is definitely getting into the weeds for me, but at least it's something to further investigate.

[jlaine]

Thanks for this, it's definitely narrowing down the scope. What puzzles me is that the code generating seems related to an output container, meaning we're writing a media file. Unless I'm mistaken the only place this occurs would be if MediaRecorder is being used, which is not the case in the webcam example.

@W3AXL what example are you running?

[W3AXL]

This is part of a larger project I'm working on, but the WebRTC functionality is relatively straightforward. Basically, I'm just doing bidirectional audio between a javascript client and the python server. Single audio track in each direction, nothing more.

The entirety of the aiortc code in my project is located on these lines: https://github.com/W3AXL/python-radio-console/blob/version-2/radio-daemon.py#L391-L535.

I'm not doing anything super funny or out of the ordinary as far as I can tell.

[jlaine]

@W3AXL can you try replacing MediaRecorder with MediaBlackhole please and re-running your measurement?

[W3AXL]

Replaced the one instance of MediaRecorder with MediaBlackhole in the above mentioned code. Pretty noticable improvement - the script runs continuously now even with jemalloc profiling enabled. The av_malloc usage is still there, but considerably lower than before. This is after a little over an hour of runtime vs the 30 minutes from the previous profiling.

Total: 16488306 B
 8191040  49.7%  49.7%  8191040  49.7% av_malloc@@LIBAVUTIL_56
 7586874  46.0%  95.7%  7586874  46.0% prof_backtrace_impl
  375552   2.3%  98.0%   729704   4.4% _dl_rtld_di_serinfo@@GLIBC_PRIVATE
   95360   0.6%  98.5%    95360   0.6% asn1_perror@@LIBTASN1_0_3
   71104   0.4%  99.0%    71104   0.4% _dl_debug_state@@GLIBC_PRIVATE
   65600   0.4%  99.4%    66864   0.4% snd_shm_area_destructor@@ALSA_0.9
   24576   0.1%  99.5%    24704   0.1% av_buffer_realloc@@LIBAVUTIL_56
   12840   0.1%  99.6%    12840   0.1% g_malloc0
   12800   0.1%  99.7%    12800   0.1% pixman_image_get_format
   11392   0.1%  99.8%    11392   0.1% allocate_owning_object (inline)

[jlaine]

I'm confused here: are you actually talking about the same issue as the original poster? Is it just "aiortc/pyav uses memory", or "aiortc/pyav has a runaway memory usage which is continuously increasing"?

[W3AXL]

It's the same issue - with MediaRecorder in the above-mentioned program, memory usage runs away, exceeding the memory available in my VM after a few days. Replacing that with a MediaBlackhole resulted in almost no memory increase over the same timespan.

[jlaine]

OK so it sounds as though there is something wonky in PyAV's memory ownership model for packets.. We probably need a much simpler test case which just runs packets through PyAV, I don't think aiortc's the source of the problem.

[W3AXL]

I'd agree with you, it looks like the _container.mux(packet) function call is the last stop for the memory leak chain inside aiortc. To my untrained eye it doesn't appear you're doing anything wrong in your calls to PyAV either. It's interesting though that there seems to be nobody else with an issue like this on the PyAV issue tracker, besides a single issue from 2018 which was allegedly fixed.

I could go ahead and open an issue over there, but since aiortc is not my library I'll be of limited use if they start asking too many low-level questions.

Your responses and willingness to help me with this issue are greatly appreciated!

[W3AXL]

Just for kicks, I upgraded my PyAV version to 10.0.0 to see if the issue still exists.

[enard-dbzr]

I had the same problem when I tried to pass the MediaRelay instead of the original MediaStream to the MediaRecorder

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[david-goldb]

I had the same problem when I tried to pass the MediaRelay instead of the original MediaStream to the MediaRecorder

@enard-dbzr We have a similar problem. Did you find a solution to this issue?

[martin-hunt]

My application is crashing due to high memory usage. It appears to be due to MediaPlayer

After some investigation, this simple code demonstrates the problem. Memory usage increases with time despite playing the same file over and over.

async def test_av(filename):
    player = MediaPlayer(filename)
    cnt = 0
    while True:
        try:
            af, vf = await asyncio.gather(player.audio.recv(), player.video.recv())
        except Exception as e:
            print(f'Error: {e}')
            break
        # print(f'{cnt}: {af} {vf}')
        cnt += 1

for _ in range(100):
    asyncio.run(test_av('video.mp4'))

[PerryDP]

Version aiortc 1.8.0
Using mediaplayer to load video files, if executed only once, everything will be fine
When there is a need for commutation, reinitialize a mediaplayer
current_video_player=MediaPlayer (video_path, loop=True)

for sender in pc. getSenders():
    if sender. track. kind=="video":
        sender. replaceTrack (current_video_player. video)
    If sender. track. kind=="audio":
        sender. replaceTrack (current_video_player. audio)

Afterwards, there will be a surge in memory usage
Even if I manually delete the old media player and perform garbage collection, it still cannot be avoided

[jlaine]

My application is crashing due to high memory usage. It appears to be due to MediaPlayer

After some investigation, this simple code demonstrates the problem. Memory usage increases with time despite playing the same file over and over.

async def test_av(filename):
    player = MediaPlayer(filename)
    cnt = 0
    while True:
        try:
            af, vf = await asyncio.gather(player.audio.recv(), player.video.recv())
        except Exception as e:
            print(f'Error: {e}')
            break
        # print(f'{cnt}: {af} {vf}')
        cnt += 1

for _ in range(100):
    asyncio.run(test_av('video.mp4'))

If you never close the tracks, the media player's memory is not released.

player.audio.stop()
player.video.stop()