[youtube] hangs with HLS format

[shaicoleman]

• I've verified and I assure that I'm running youtube-dl 2016.04.06
• At least skimmed through README and most notably FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)

When downloading a youtube video with HLS, it just hangs

$ youtube-dl -v -f 94 https://www.youtube.com/watch?v=Gjv9LrRPo0Q

[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'-f', u'94', u'https://www.youtube.com/watch?v=Gjv9LrRPo0Q']
[debug] Encodings: locale ANSI_X3.4-1968, fs ANSI_X3.4-1968, out None, pref ANSI_X3.4-1968
[debug] youtube-dl version 2016.04.06
[debug] Python version 2.7.11+ - Linux-4.4.0-14-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.6-1ubuntu2, ffprobe 2.8.6-1ubuntu2
[debug] Proxy map: {}
[youtube] Gjv9LrRPo0Q: Downloading webpage
[youtube] Gjv9LrRPo0Q: Downloading video info webpage
[youtube] Gjv9LrRPo0Q: Extracting video information
[youtube] Gjv9LrRPo0Q: Downloading formats manifest
[youtube] Gjv9LrRPo0Q: Downloading MPD manifest
[debug] Invoking downloader on u'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Gjv9LrRPo0Q.0/itag/94/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/itag%3D140/sgovp/itag%3D135/hls_chunk_host/r13---sn-q0c7dn7y.googlevideo.com/gcr/ie/playlist_type/CLEAN/mm/32/mn/sn-q0c7dn7y/ms/lv/mv/m/nh/IgpwcjAxLmR1YjA2Kg0yMTMuNDYuMTY1LjUz/pl/20/dover/3/upn/sE2WEHyl5fo/fexp/9416126,9416891,9420452,9422596,9423662,9426926,9427902,9428398,9431865/sver/3/mt/1460041212/ip/89.101.24.177/ipbits/0/expire/1460062850/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,gcr,playlist_type,mm,mn,ms,mv,nh,pl/signature/575B0422A38266D6543B3AB9BBA717D9BE02DE21.5DD5ABCD13415E0C8A885DB720C2C4B1101B85FF/key/dg_yt0/playlist/index.m3u8'
[download] Destination: Live video-Gjv9LrRPo0Q.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/44.0 (Chrome)
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
' -i https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Gjv9LrRPo0Q.0/itag/94/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/itag%3D140/sgovp/itag%3D135/hls_chunk_host/r13---sn-q0c7dn7y.googlevideo.com/gcr/ie/playlist_type/CLEAN/mm/32/mn/sn-q0c7dn7y/ms/lv/mv/m/nh/IgpwcjAxLmR1YjA2Kg0yMTMuNDYuMTY1LjUz/pl/20/dover/3/upn/sE2WEHyl5fo/fexp/9416126,9416891,9420452,9422596,9423662,9426926,9427902,9428398,9431865/sver/3/mt/1460041212/ip/89.101.24.177/ipbits/0/expire/1460062850/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,gcr,playlist_type,mm,mn,ms,mv,nh,pl/signature/575B0422A38266D6543B3AB9BBA717D9BE02DE21.5DD5ABCD13415E0C8A885DB720C2C4B1101B85FF/key/dg_yt0/playlist/index.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc 'file:Live video-Gjv9LrRPo0Q.mp4.part'
ffmpeg version 2.8.6-1ubuntu2 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.1 (Ubuntu 5.3.1-11ubuntu1) 20160311
  configuration: --prefix=/usr --extra-version=1ubuntu2 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-opengl --enable-x11grab --enable-libdc1394 --enable-libiec61883 --enable-libzmq --enable-frei0r --enable-libx264 --enable-libopencv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, hls,applehttp, from 'https://manifest.googlevideo.com/api/manifest/hls_playlist/id/Gjv9LrRPo0Q.0/itag/94/source/yt_live_broadcast/requiressl/yes/ratebypass/yes/live/1/cmbypass/yes/goi/160/sgoap/itag%3D140/sgovp/itag%3D135/hls_chunk_host/r13---sn-q0c7dn7y.googlevideo.com/gcr/ie/playlist_type/CLEAN/mm/32/mn/sn-q0c7dn7y/ms/lv/mv/m/nh/IgpwcjAxLmR1YjA2Kg0yMTMuNDYuMTY1LjUz/pl/20/dover/3/upn/sE2WEHyl5fo/fexp/9416126,9416891,9420452,9422596,9423662,9426926,9427902,9428398,9431865/sver/3/mt/1460041212/ip/89.101.24.177/ipbits/0/expire/1460062850/sparams/ip,ipbits,expire,id,itag,source,requiressl,ratebypass,live,cmbypass,goi,sgoap,sgovp,hls_chunk_host,gcr,playlist_type,mm,mn,ms,mv,nh,pl/signature/575B0422A38266D6543B3AB9BBA717D9BE02DE21.5DD5ABCD13415E0C8A885DB720C2C4B1101B85FF/key/dg_yt0/playlist/index.m3u8':
  Duration: N/A, start: 5053.916000, bitrate: N/A
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 127 kb/s
    Stream #0:1: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 854x474 [SAR 1:1 DAR 427:237], 25 fps, 25 tbr, 90k tbn, 50 tbc
[mp4 @ 0x1fce660] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x1fce660] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to 'file:Live video-Gjv9LrRPo0Q.mp4.part':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 854x474 [SAR 1:1 DAR 427:237], q=2-31, 25 fps, 25 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 127 kb/s
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  341 fps=0.0 q=-1.0 size=    2192kB time=00:00:13.66 bitrate=1313.4kbits/s

[yan12125]

For developers: This video has DASH formats and youtube-dl should be able to process them.

[yan12125]

@remitamine Simply removing the following check works for this video:

diff --git a/youtube_dl/extractor/common.py b/youtube_dl/extractor/common.py
index ec6625e..6794b07 100644
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -1401,9 +1401,6 @@ class InfoExtractor(object):
             compat_etree_fromstring(mpd.encode('utf-8')), mpd_id, mpd_base_url, formats_dict=formats_dict)

     def _parse_mpd_formats(self, mpd_doc, mpd_id=None, mpd_base_url='', formats_dict={}):
-        if mpd_doc.get('type') == 'dynamic':
-            return []
-
         namespace = self._search_regex(r'(?i)^{([^}]+)?}MPD$', mpd_doc.tag, 'namespace', default=None)

         def _add_ns(path):

Is this check really necessary?

[remitamine]

there was a previous discussion with @dstftw about this(i don't remember on what commit).
the problem is that if we enable dynamic type the user will expect it to download the live stream like what ffmpeg does for HLS streams but the DASH downloader didn't support downloading live streams(updating the MPD, sync the video and the audio, check for segment availability...).
i have some code to move the code to parse the MPD to the downloader(needs more tests) that will help in adding support for live streams.

[dstftw]

This is to skip livestreams since we are unable to process real livestreams with audio/video separated. This video however does not look like livestream however manifest tells it is.

[yan12125]

I see the rationale. I guess it's because YouTube is still in progress of merging live stream segments into a single file for each format. Maybe we can detect whether it's really a live stream or not, and enable DASH formats for those just finished their live events?

[yan12125]

YouTube has done post-processing for Gjv9LrRPo0Q. Superseded by #10787.