Ubuntu 16.04: MemoryError #9306

ilf · 2016-04-25T09:15:54Z

Since the upgrade to Ubuntu 16.04, youtube-dl eats a lot of memory and then crashes.

% youtube-dl --version
2016.04.24

% python --version
Python 2.7.11+

% youtube-dl "XXXXXXXXXXX"
[youtube] XXXXXXXXXXX: Downloading webpage
[youtube] XXXXXXXXXXX: Downloading video info webpage
[youtube] XXXXXXXXXXX: Extracting video information
[youtube] XXXXXXXXXXX: Downloading MPD manifest
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 419, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 409, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1732, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 684, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 729, in process_ie_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1361, in process_video_result
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 48, in __init__
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 105, in _determine_executables
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 105, in <genexpr
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 1621, in get_exe_version
MemoryError

yan12125 · 2016-04-25T10:25:37Z

From the log youtube-dl fails when trying to decode the output of one of the following commands: ffmpeg -version, avconv -version, ffprobe -version, avprobe -version. All of these commands print a few lines of messages (something like several KBytes), so this line shouldn't eats all of memory.

% python --version
Python 2.7.11+

Seems it's not authentic Python 2.7.11. (Usually a plus sign indicates "modified") Probably a Ubuntu issue. Please report it to them.

ilf · 2016-04-25T10:39:18Z

% ffmpeg -version
ffmpeg version 2.8.6-1ubuntu2 Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 5.3.1 (Ubuntu 5.3.1-11ubuntu1) 20160311
configuration: --prefix=/usr --extra-version=1ubuntu2 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite
--enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100

/usr/bin/avconv is a symlink to /usr/bin/ffmpeg

% ffprobe -version
ffprobe version 2.8.6-1ubuntu2 Copyright (c) 2007-2016 the FFmpeg developers
built with gcc 5.3.1 (Ubuntu 5.3.1-11ubuntu1) 20160311
configuration: --prefix=/usr --extra-version=1ubuntu2 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite
--enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100

/usr/bin/avprobe is a Symlink to /usr/bin/ffprobe

Trying to monitor it with top, it does indeed seem like avprobe is eating a lot of CPU (98 %) and RAM (30 %) there. youtube-dl also uses another 68% CPU and 8.5 % RAM.

% apt-cache policy python
python:
  Installed: 2.7.11-1

Is it really an issue with the Ubuntu Python package? If so, what do I report to them?

yan12125 · 2016-04-25T10:57:06Z

What's your total memory size?

ilf · 2016-04-25T11:01:42Z

4 GB

yan12125 · 2016-04-25T11:12:31Z

avprobe eats 1.2GB. It's unusual. I'm not sure whether it's a ffprobe or Python issue. Maybe both.

Could you try with this Python script:

import subprocess

out, _ = subprocess.Popen(
    ['avprobe', '-version'],
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()
out = out.decode('ascii', 'ignore')
print(out)

It's a minimal version extracted from the failing point in youtube-dl.

ilf · 2016-04-25T12:10:53Z

Works:

% time python ./test.py
ffprobe version 2.8.6-1ubuntu2 Copyright (c) 2007-2016 the FFmpeg developers
built with gcc 5.3.1 (Ubuntu 5.3.1-11ubuntu1) 20160311
configuration: --prefix=/usr --extra-version=1ubuntu2 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite
--enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
libavutil      54. 31.100 / 54. 31.100
libavcodec     56. 60.100 / 56. 60.100
libavformat    56. 40.101 / 56. 40.101
libavdevice    56.  4.100 / 56.  4.100
libavfilter     5. 40.101 /  5. 40.101
libavresample   2.  1.  0 /  2.  1.  0
libswscale      3.  1.101 /  3.  1.101
libswresample   1.  2.101 /  1.  2.101
libpostproc    53.  3.100 / 53.  3.100

python ./test.py  0.05s user 0.02s system 93% cpu 0.069 total

yan12125 · 2016-04-25T13:36:04Z

I just tried on Ubuntu 16.04 live DVD and everything works. I'm afraid I can't help more. Ubuntu places may give you more help.

OverlordQ · 2016-05-16T23:07:16Z

Are you trying to pull down videos through Tor and trying to mux them at the same time? I get this error and this is due to torsocks being incompatible with ffmpeg. Doing an strace on the process you see:

read(3, "[May 16 19:03:47] WARNING torsocks[3038]: [syscall] Unsupported syscall number 204. Denying the call"..., 12168823) = 140
read(3, "[May 16 19:03:47] WARNING torsocks[3038]: [syscall] Unsupported syscall number 204. Denying the call"..., 12168683) = 140
read(3, "[May 16 19:03:47] WARNING torsocks[3038]: [syscall] Unsupported syscall number 204. Denying the call"..., 12168543) = 140
read(3, "[May 16 19:03:47] WARNING torsocks[3038]: [syscall] Unsupported syscall number 204. Denying the call"..., 12168403) = 140

Until it runs out of memory.

yan12125 · 2016-05-17T02:42:08Z

It's a known bug (#402 (comment)). Note that SOCKS proxies support is available since version 2016.05.10. Use --proxy socks5://ip:port to enable it.

OverlordQ · 2016-05-17T04:25:03Z

Yeah, I was just making sure that wasn't the particular issue they were running into.

ilf · 2016-05-17T14:01:49Z

Oh yeah, I am using Tor, this was the issue. Sorry for the noise.

yan12125 added the cant-reproduce label Apr 25, 2016

ilf closed this May 17, 2016

yan12125 removed the cant-reproduce label May 17, 2016

ytdl-org / youtube-dl

Ubuntu 16.04: MemoryError #9306

Ubuntu 16.04: MemoryError #9306

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

OverlordQ commented May 16, 2016 •

edited

yan12125 commented May 17, 2016

OverlordQ commented May 17, 2016

ilf commented May 17, 2016

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

Ubuntu 16.04: MemoryError #9306

Ubuntu 16.04: MemoryError #9306

Comments

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

ilf commented Apr 25, 2016

yan12125 commented Apr 25, 2016

OverlordQ commented May 16, 2016 • edited

yan12125 commented May 17, 2016

OverlordQ commented May 17, 2016

ilf commented May 17, 2016

Essential cookies

Always active

Analytics cookies

OverlordQ commented May 16, 2016 •

edited