Help with trimming, scaling and concatenating from one input file (memory leak)

[leferi99]

Hello all,

I scoured forums (and issues here) for some time and couldn't find the cause of my problem so I'm opening an issue. For some reason my implementation of a video trimming, downscaling and concatenating script results in a memory leak crashing ffmpeg, Python, and sometimes even my system. See (hopefully all) the relevant code below:

input_stream = ffmpeg.input(input_path)
starts = listoftimes[:, 0]
ends = listoftimes[:, 1]
vid = input_stream.video
aud = input_stream.audio
streams = []

for i in range(len(listoftimes)):
    streams.append(vid.trim(start=int(starts[i]), end=int(ends[i])).setpts('PTS-STARTPTS'))
    streams.append(aud.filter_('atrim', start=int(starts[i]), end=int(ends[i])).filter_('asetpts', 'PTS-STARTPTS'))
joined = ffmpeg.concat(*streams, v=1, a=1).node
output = ffmpeg.output(joined[0], joined[1], output_path).overwrite_output()
output.run()

listoftimes contains the starts and ends of the to-be-trimmed parts in seconds. When ffmpeg didn't crash my system it wrote some error like "Cannot allocate memory". I will try to get the logs but basic information of the video file: mp4, h264, ~35 minutes, 3Mbps, 1280x708. I would trim this down to around 12 minutes from lots of short, hand-selected parts. I was thinking maybe ffmpeg opens the input just as many times as many parts I choose, but since I'm a beginner with ffmpeg and ffmpeg-python I don't quite see what I should change in my code for this to not occur.

I would appreciate some help or ideas on what I might be doing wrong. I will be back with the ffmpeg logs if I can just almost crash my PC.

[leferi99]

Okay so I tried it with another video file, this time without scaling and the same thing happens, here are the logs:

ffmpeg version 6.1-full_build-www.gyan.dev Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.0 (Rev10, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --pkg-config=pkgconf --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-dxva2 --enable-d3d11va --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
  libpostproc    57.  3.100 / 57.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '../../v/test2.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:57:33.79, start: 0.000000, bitrate: 2002 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1280x720, 1789 kb/s, 23.98 fps, 23.98 tbr, 90k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 209 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:0 (h264) -> trim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  Stream #0:1 (aac) -> atrim:default
  concat -> Stream #0:0 (libx264)
  concat -> Stream #0:1 (aac)
Press [q] to stop, [?] for help
[libx264 @ 0000026a944dccc0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0000026a944dccc0] profile High, level 3.1, 4:2:0, 8-bit
[libx264 @ 0000026a944dccc0] 264 - core 164 r3161 a354f11 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=22 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=23 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to '../../out/test2.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf60.16.100
  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 1280x720, q=2-31, 23.98 fps, 24k tbn
    Metadata:
      encoder         : Lavc60.31.102 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
  Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc60.31.102 aac
frame= 3523 fps=153 q=28.0 size=   23296kB time=00:02:30.39 bitrate=1268.9kbits/s speed=6.54x

After this point it ate all 16GB of RAM, then the swap file quickly grew to 50GB+, the speed decreased to 1.2x, the system became slow to respond and I tried to shut down the kernel, and I succeeded.

[vost#0:0/libx264 @ 0000026a92dad8c0] Error submitting a packet to the muxer: Immediate exit requested.957x
[out#0/mp4 @ 0000026a92dc9c00] Error muxing a packet
[I 2024-01-03 00:01:07.315 ServerApp] Interrupted...
[out#0/mp4 @ 0000026a92dc9c00] Error writing trailer: Immediate exit requested
[out#0/mp4 @ 0000026a92dc9c00] Error closing file: Immediate exit requested
[out#0/mp4 @ 0000026a92dc9c00] video:28349kB audio:3178kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame= 4718 fps= 22 q=-1.0 Lsize=   31232kB time=00:03:18.99 bitrate=1285.7kbits/s dup=1 drop=0 speed=0.945x
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
(work) PS C:\Users\lefer> [libx264 @ 0000026a944dccc0] frame I:31    Avg QP:16.53  size: 63630
[libx264 @ 0000026a944dccc0] frame P:1528  Avg QP:20.64  size: 11423
[libx264 @ 0000026a944dccc0] frame B:3213  Avg QP:24.12  size:  3124
[libx264 @ 0000026a944dccc0] consecutive B-frames:  5.2%  6.1% 27.0% 61.7%
[libx264 @ 0000026a944dccc0] mb I  I16..4: 14.4% 55.2% 30.4%
[libx264 @ 0000026a944dccc0] mb P  I16..4:  7.6% 19.6%  0.8%  P16..4: 29.2%  6.9%  3.0%  0.0%  0.0%    skip:32.8%
[libx264 @ 0000026a944dccc0] mb B  I16..4:  1.2%  2.4%  0.0%  B16..8: 29.4%  2.3%  0.3%  direct: 1.0%  skip:63.4%  L0:53.0% L1:43.9% BI: 3.1%
[libx264 @ 0000026a944dccc0] 8x8 transform intra:68.3% inter:83.1%
[libx264 @ 0000026a944dccc0] coded y,uvDC,uvAC intra: 31.8% 55.6% 6.6% inter: 5.6% 11.3% 0.2%
[libx264 @ 0000026a944dccc0] i16 v,h,dc,p: 35% 23% 15% 27%
[libx264 @ 0000026a944dccc0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 17% 26%  3%  4%  6%  3%  5%  2%
[libx264 @ 0000026a944dccc0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 18% 12%  5%  8% 11%  6%  7%  5%
[libx264 @ 0000026a944dccc0] i8c dc,h,v,p: 51% 20% 25%  4%
[libx264 @ 0000026a944dccc0] Weighted P-Frames: Y:1.0% UV:0.7%
[libx264 @ 0000026a944dccc0] ref P L0: 74.1%  9.9% 12.3%  3.6%  0.0%
[libx264 @ 0000026a944dccc0] ref B L0: 89.7%  9.2%  1.1%
[libx264 @ 0000026a944dccc0] ref B L1: 97.4%  2.6%
[libx264 @ 0000026a944dccc0] kb/s:1184.37
[aac @ 0000026a944dd4c0] Qavg: 723.430

At this point as an alternative I'm thinking maybe I should export the trimmed parts into temporary files and maybe use concat demuxer to put them together again or what would be fastest?

[leferi99]

Okay, so I went with the following: first trim (with seeking) and encode the parts, then use the contact demuxer in a different step to put it all together. Still, if it could be done in one step without the memory leak, it would be very nice, so I'll keep this issue open. Someone might have a suggestion.

[welteki]

I'm also experiencing this memory leak.

My script started failing with the latest ffmpeg release 6.1 but works with older ffmpeg versions.

I run a very similar workflow trim -> concat -> scale from a single input file:

def generate_video_preview(in_filename, out_filename, sample_duration, sample_seconds, scale, format, quiet):
    in_file = ffmpeg.input(in_filename)

    samples = sample_video(in_file, sample_duration=sample_duration, sample_seconds=sample_seconds)
    stream = ffmpeg.concat(*samples)

    if scale is not None:
        width, height = scale.split(':')
        stream = ffmpeg.filter(stream, 'scale', width=width, height=height, force_original_aspect_ratio='decrease')

    (
        ffmpeg
        .output(stream, out_filename, format=format)
        .overwrite_output()
        .run(quiet=quiet)
    )

[leferi99]

See I was sure that earlier when I tried this it worked without a memory leak, but by earlier I mean a couple of weeks ago. Then I changed computers and the exact same thing produced a memory leak. So it must have been the version change. But I would say it was maybe rather 6.1.1 not the 6.1 release. Unfortunately I cannot test on my old computer anymore since it has been wiped.

[Gistix]

I have found the reason for the memory leak on my end.

Make sure start and end are actual timestamps or a floating point seconds format, I was using start_frame/end_frame for both trim and filter's atrim which caused these leaks.