Memory leak program on big Youtube playlist

[damamasss]

When I download big playlist (more than 500 videos) from Youtube with --datebefore (may be and without datebefore), on Debian 8 machine, youtube-dl have leak memory - full 4G RAM is occupied by the youtube-dl. And PC is frozen afret ~450 skipped videos

/usr/local/bin/youtube-dl -v -ci -x --audio-format mp3 --audio-quality 128K --datebefore 20170329 -o '%(upload_date)s-%(uploader)s-%(title)s-%(id)s.%(ext)s' https://www.youtube.com/channel/UCTwUWzrI3iAxQwzbj5yJsvw/videos
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-ci', u'-x', u'--audio-format', u'mp3', u'--audio-quality', u'128K', u'--datebefore', u'20170329', u'-o', u'%(upload_date)s-%(uploader)s-%(title)s-%(id)s.%(ext)s', u'https://www.youtube.com/channel/UCTwUWzrI3iAxQwzbj5yJsvw/videos']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.07.30.1
[debug] Python version 2.7.9 - Linux-3.16.0-4-amd64-x86_64-with-debian-8.9
[debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9
[debug] Proxy map: {}
[youtube:channel] UCTwUWzrI3iAxQwzbj5yJsvw: Downloading channel page
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading webpage
[download] Downloading playlist: Uploads from БМ Знания
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #1
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #2
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #3
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #4
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #5
[youtube:playlist] playlist Uploads from БМ Знания: Downloading 545 videos
[download] Downloading video 1 of 545
[youtube] F0Ldm10xbn4: Downloading webpage
[youtube] F0Ldm10xbn4: Downloading video info webpage
[youtube] F0Ldm10xbn4: Extracting video information
[youtube] F0Ldm10xbn4: Downloading MPD manifest
[youtube] F0Ldm10xbn4: Downloading MPD manifest
[download] 2017-08-01 upload date is not in range 0001-01-01 - 2017-03-29
[download] Downloading video 2 of 545
[youtube] x1ryKOZU0DY: Downloading webpage
[youtube] x1ryKOZU0DY: Downloading video info webpage
[youtube] x1ryKOZU0DY: Extracting video information
[youtube] x1ryKOZU0DY: Downloading MPD manifest
[youtube] x1ryKOZU0DY: Downloading MPD manifest
[download] 2017-08-01 upload date is not in range 0001-01-01 - 2017-03-29
[download] Downloading video 3 of 545
[youtube] AWXrhlPMvFM: Downloading webpage
^C
ERROR: Interrupted by user

[damamasss]

With download we also see same

...
[download] Downloading video 369 of 550
[youtube] fl5C8LIR1Zk: Downloading webpage
[youtube] fl5C8LIR1Zk: Downloading video info webpage
[youtube] fl5C8LIR1Zk: Extracting video information
[youtube] fl5C8LIR1Zk: Downloading MPD manifest
[youtube] fl5C8LIR1Zk: Downloading MPD manifest
[dashsegments] Total fragments: 393
[download] Destination: 20170424-БМ Знания-Строим Самолёт. Учеба в Бизнес Молодости. Работа в команде. «Любой может стать любым» [БМ]-fl5C8LIR1Zk.m4a
[download] 100% of 29.67MiB in 00:37
WARNING: fl5C8LIR1Zk: writing DASH m4a. Only some players support this container. Install ffmpeg or avconv to fix this automatically.
ERROR: WARNING: unable to obtain file audio codec with ffprobe
...

[damamasss]

But my ffmpeg can convert m4a

[dstftw]

There is no memory leak. High memory consumption is due to lots of DASH segments formats that carry fragments metafield. fragments metafield may be quite heavy (ranging from 0.5 to 15 MiB per video, see --write-info-json) since it contains a list of all fragments of a DASH video. Due to a high number of such videos collected metadata grows.
You can use --playlist-start and --playlist-end to process playlist in parts.

[ivan]

youtube-dl/youtube_dl/YoutubeDL.py

Line 1007 in 7c24a58

playlist_results.append(entry_result)

is responsible for sometimes retaining > 2.5GB of objects when downloading very long playlists