Skip to content

/ youtube-dl

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HLS vulnerability in both FFMpeg and NativeHlsFD #8227

Closed
yan12125 opened this issue Jan 13, 2016 · 3 comments
Closed

HLS vulnerability in both FFMpeg and NativeHlsFD #8227

yan12125 opened this issue Jan 13, 2016 · 3 comments
Labels
bug

Comments

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Jan 13, 2016

A recent post points out that FFMpeg can cause contents of arbitrary files (for example /etc/passwd) being accessible on the Internet via a malicious input file. [1][2] The reaction of Arch Linux developers is disabling affected components before they are fixed. [3] In this commit, concat: protocol and HLS support are disabled. The former one is not used in the mainline codebase, just in some pull requests (#2844). The latter one is more serious. I've just updated my copy to the latest official Arch binary. Downloading an YouTube live stream gives:

$ youtube-dl -v "https://www.youtube.com/watch?v=clO8XxFkrj4"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'https://www.youtube.com/watch?v=clO8XxFkrj4']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.09
[debug] Git HEAD: 40cf7fc
[debug] Python version 3.5.1 - Linux-4.3.3-2-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 2.8.4, ffprobe 2.8.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] clO8XxFkrj4: Downloading webpage
[youtube] clO8XxFkrj4: Downloading video info webpage
[youtube] clO8XxFkrj4: Extracting video information
[youtube] clO8XxFkrj4: Downloading formats manifest
[youtube] clO8XxFkrj4: Downloading DASH manifest
[debug] Invoking downloader on 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8'
[download] Destination: 中天電視直播HD頻道(總統大選直播區如下) │Taiwan CTITV News HD Live-clO8XxFkrj4.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
' -i https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8 -f mp4 -c copy -bsf:a aac_adtstoasc 'file:中天電視直播HD頻道(總統大選直播區如下) │Taiwan CTITV News HD Live-clO8XxFkrj4.mp4.part'
ffmpeg version 2.8.4 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab --disable-demuxer=hls --disable-protocol='concat,hls'
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8: Invalid data found when processing input


ERROR: ffmpeg exited with code 1
  File "/usr/bin/youtube-dl", line 9, in <module>
    load_entry_point('youtube-dl==2016.1.9', 'console_scripts', 'youtube-dl')()
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/__init__.py", line 410, in main
    _real_main(argv)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/__init__.py", line 400, in _real_main
    retcode = ydl.download(all_urls)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1677, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 676, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 722, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1347, in process_video_result
    self.process_info(new_info)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1609, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 1551, in dl
    return fd.download(name, info)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/downloader/common.py", line 342, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/downloader/hls.py", line 63, in real_download
    self.report_error('%s exited with code %d' % (ffpp.basename, retval))
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/downloader/common.py", line 155, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 540, in report_error
    self.trouble(error_message, tb)
  File "/home/yen/Executables/Multimedia/youtube-dl/youtube_dl/YoutubeDL.py", line 502, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

Running the command directly does not give more information:

$ ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/20.0 (Chrome)
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
' -i https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8 -f mp4 -c copy -bsf:a aac_adtstoasc 'file:中天電視直播HD頻道(總統大選直播區如下) │Taiwan CTITV News HD Live-clO8XxFkrj4.mp4.part'
ffmpeg version 2.8.4 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab --disable-demuxer=hls --disable-protocol='concat,hls'
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[https @ 0x55f0db7dfac0] No trailing CRLF found in HTTP header.
https://manifest.googlevideo.com/api/manifest/hls_playlist/id/clO8XxFkrj4.2/itag/95/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/gir/yes/dg_shard/Y2xPOFh4RmtyajQuMg.95/hls_chunk_host/r5---sn-5njj-u2xl.googlevideo.com/playlist_type/LIVE/pmbypass/yes/gcr/tw/mm/32/mn/sn-5njj-u2xl/ms/lv/mv/m/pl/16/dover/3/fexp/9416126,9420452,9422596,9423459,9423662,9427015/upn/96epxf2PvxI/sver/3/mt/1452716770/ip/140.112.230.216/ipbits/0/expire/1452738400/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,gir,dg_shard,hls_chunk_host,playlist_type,pmbypass,gcr,mm,mn,ms,mv,pl/signature/483695472960B09D033F3C6FA6C5E2BB5769C122.93B057124DC884D539E8ED21C5A6CBE825B69EBB/key/dg_yt0/playlist/index.m3u8: Invalid data found when processing input

Before the problem fixed or Arch developers decided to have a different workaround, the only way is suggesting --hls-native-native. However, NativeHlsFD is also vulnerable. With the following evil.m3u8:

#EXTM3U
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10.0
file:///etc/passwd
#EXT-X-ENDLIST

youtube-dl gives undesired results:

$ youtube-dl -v --hls-prefer-native http://localhost/yen/test/evil.m3u8
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '--hls-prefer-native', 'http://localhost/yen/test/evil.m3u8']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.09
[debug] Git HEAD: 40cf7fc
[debug] Python version 3.5.1 - Linux-4.3.3-2-ARCH-x86_64-with-arch-Arch-Linux
[debug] exe versions: ffmpeg 2.8.4, ffprobe 2.8.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] evil: Requesting header
WARNING: Falling back on generic information extractor.
[generic] evil: Downloading webpage
WARNING: URL could be a direct video link, returning it as such.
[debug] Invoking downloader on 'http://localhost/yen/test/evil.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 1
[download] Destination: evil-evil.m3u8
[download] 100% of 2.25KiB in 00:00

$ head -n 3 evil-evil.m3u8 
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/bin/false
daemon:x:2:2:daemon:/sbin:/bin/false

All servers using youtube-dl are affected, whether they use FFMpeg or NativeHlsFD. I guess only http:, https: and data: protocols are necessary for YoutubeDL.urlopen()? We should check URLs before passing them into urllib2.

[1] https://news.ycombinator.com/item?id=10893301
[2] http://habrahabr.ru/company/mailru/blog/274855/ (The original post, in Russian)
[3] https://projects.archlinux.org/svntogit/packages.git/commit/trunk/PKGBUILD?h=packages/ffmpeg&id=ef0b4890e18a52e976274d02a09738f73a07f4d2
@yan12125 yan12125 added the bug label Jan 13, 2016
@jaimeMF
Copy link
Collaborator

@jaimeMF jaimeMF commented Jan 13, 2016

I've suggested a patch for disabling the 'file:' protocol in #8228. Note that actually all you need to do to get access to a sensitive file is to run youtube-dl file:///etc/passwd, you don't need to craft a m3u8 file and serve it online.

Unfortunately there doesn't seem to be a way to disable protocols on ffmpeg at runtime, which would be the simple solution.
dstftw added a commit that referenced this issue Jan 14, 2016
@dstftw
Merge pull request #8228 from jaimeMF/disable-file-handler
Loading status checks…
345f121 
[YoutubeDL] urlopen: disable the 'file:' protocol (#8227)
@Kagami
Copy link
Contributor

@Kagami Kagami commented Jan 15, 2016

Fixed in ffmpeg git master (and few more recent commits).
@yan12125 yan12125 mentioned this issue Jan 15, 2016
Closed
@yan12125
Copy link
Collaborator Author

@yan12125 yan12125 commented Jan 15, 2016

Thanks for the useful information @Kagami ! Now ffmpeg is fixed both in git-master and Arch Linux's package, and I guess other distributions will fix their packages soon. If there are more HLS vulnerability discovered, this issue can be re-opened.
@yan12125 yan12125 closed this Jan 15, 2016
@yan12125 yan12125 mentioned this issue Mar 5, 2017
Closed
@yan12125 yan12125 mentioned this issue Jul 3, 2017
Open
4 of 8 tasks complete
@tv21 tv21 mentioned this issue Aug 25, 2019
Closed
5 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
@Kagami @jaimeMF @yan12125
You can’t perform that action at this time.