HLS vulnerability in both FFMpeg and NativeHlsFD

[yan12125]

A recent post points out that FFMpeg can cause contents of arbitrary files (for example /etc/passwd) being accessible on the Internet via a malicious input file. [1][2] The reaction of Arch Linux developers is disabling affected components before they are fixed. [3] In this commit, concat: protocol and HLS support are disabled. The former one is not used in the mainline codebase, just in some pull requests (#2844). The latter one is more serious. I've just updated my copy to the latest official Arch binary. Downloading an YouTube live stream gives:

$ youtube-dl -v "https://www.youtube.com/watch?v=clO8XxFkrj4"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=clO8XxFkrj4']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.09
[debug] Git HEAD: 40cf7fc
[debug] Python version 3.5.1 - Linux-4.3.3-2-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 2.8.4, ffprobe 2.8.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] clO8XxFkrj4: Downloading webpage
[youtube] clO8XxFkrj4: Downloading video info webpage
[youtube] clO8XxFkrj4: Extracting video information
[youtube] clO8XxFkrj4: Downloading formats manifest
[youtube] clO8XxFkrj4: Downloading DASH manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8'
[download] Destination: 中天電視直播HD頻道(總統大選直播區如下) │Taiwan CTITV News HD Live-clO8XxFkrj4.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
' -i https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8 -f mp4 -c copy -bsf:a aac_adtstoasc 'file:中天電視直播HD頻道(總統大選直播區如下) │Taiwan CTITV News HD Live-clO8XxFkrj4.mp4.part'
ffmpeg version 2.8.4 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab --disable-demuxer=hls --disable-protocol='concat,hls'
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8: Invalid data found when processing input


ERROR: ffmpeg exited with code 1
  File "/usr/bin/youtube-dl", line 9, in <module>
    load_entry_point('youtube-dl==2016.1.9', 'console_scripts', 'youtube-dl')()
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/__init__.py", line 410, in main
    _real_main(argv)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/__init__.py", line 400, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1677, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 676, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 722, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1347, in process_video_result
    self.process_info(new_info)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1609, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1551, in dl
    return fd.download(name, info)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/downloader/common.py", line 342, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/downloader/hls.py", line 63, in real_download
    self.report_error('%s exited with code %d' % (ffpp.basename, retval))
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/downloader/common.py", line 155, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 540, in report_error
    self.trouble(error_message, tb)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 502, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

Running the command directly does not give more information:

$ ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
' -i https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8 -f mp4 -c copy -bsf:a aac_adtstoasc 'file:中天電視直播HD頻道(總統大選直播區如下) │Taiwan CTITV News HD Live-clO8XxFkrj4.mp4.part'
ffmpeg version 2.8.4 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab --disable-demuxer=hls --disable-protocol='concat,hls'
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[https @ 0x55f0db7dfac0] No trailing CRLF found in HTTP header.
https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8: Invalid data found when processing input

Before the problem fixed or Arch developers decided to have a different workaround, the only way is suggesting --hls-native-native. However, NativeHlsFD is also vulnerable. With the following evil.m3u8:

#EXTM3U
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10.0
file:///etc/passwd
#EXT-X-ENDLIST

youtube-dl gives undesired results:

$ youtube-dl -v --hls-prefer-native http://localhost/yen/test/evil.m3u8
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '--hls-prefer-native', 'http://localhost/yen/test/evil.m3u8']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.09
[debug] Git HEAD: 40cf7fc
[debug] Python version 3.5.1 - Linux-4.3.3-2-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 2.8.4, ffprobe 2.8.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] evil: Requesting header
WARNING: Falling back on generic information extractor.
[generic] evil: Downloading webpage
WARNING: URL could be a direct video link, returning it as such.
[debug] Invoking downloader on 'http://localhost/yen/test/evil.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1
[download] Destination: evil-evil.m3u8
[download] 100% of 2.25KiB in 00:00

$ head -n 3 evil-evil.m3u8 
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/bin/false
daemon:x:2:2:daemon:/sbin:/bin/false

All servers using youtube-dl are affected, whether they use FFMpeg or NativeHlsFD. I guess only http:, https: and data: protocols are necessary for YoutubeDL.urlopen()? We should check URLs before passing them into urllib2.

[1] https://news.ycombinator.com/item?id=10893301
[2] http://habrahabr.ru/company/mailru/blog/274855/ (The original post, in Russian)
[3] https://projects.archlinux.org/svntogit/packages.git/commit/trunk/PKGBUILD?h=packages/ffmpeg&id=ef0b4890e18a52e976274d02a09738f73a07f4d2

[jaimeMF]

I've suggested a patch for disabling the 'file:' protocol in #8228. Note that actually all you need to do to get access to a sensitive file is to run youtube-dl file:///etc/passwd, you don't need to craft a m3u8 file and serve it online.

Unfortunately there doesn't seem to be a way to disable protocols on ffmpeg at runtime, which would be the simple solution.

[Kagami]

Fixed in ffmpeg git master (and few more recent commits).

[yan12125]

Thanks for the useful information @Kagami ! Now ffmpeg is fixed both in git-master and Arch Linux's package, and I guess other distributions will fix their packages soon. If there are more HLS vulnerability discovered, this issue can be re-opened.