Support for #EXT-X-BYTERANGE in native hls extractor #10955

TehBotolSosro · 2016-10-18T09:20:05Z

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like that [x])
Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2016.10.16. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[ x] I've verified and I assure that I'm running youtube-dl 2016.10.16

Before submitting an issue make sure you have:

[x ] At least skimmed through README and most notably FAQ and BUGS sections
[x ] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

Bug report (encountered problems with youtube-dl)
Site support request (request for adding support for a new site)
Feature request (request for a new functionality)
Question
Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl -v https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/encryption/bad/bad.m3u8
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/encryption/bad/bad.m3u8']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2016.10.16
[debug] Python version 3.4.4 - Windows-10-10.0.10240
[debug] exe versions: ffmpeg N-82003-g9082603, ffprobe N-82003-g9082603
[debug] Proxy map: {}
[generic] bad: Requesting header
[generic] bad: Downloading m3u8 information
[debug] Invoking downloader on 'https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/encryption/bad/bad.m3u8'
[download] Destination: bad-bad.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Language: en-us,en;q=0.5
' -i https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/encryption/bad/bad.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc file:bad-bad.mp4.part
ffmpeg version N-82003-g9082603 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-libebur128 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 32.100 / 55. 32.100
  libavcodec     57. 61.103 / 57. 61.103
  libavformat    57. 52.100 / 57. 52.100
  libavdevice    57.  0.102 / 57.  0.102
  libavfilter     6. 64.100 /  6. 64.100
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  2.100 /  2.  2.100
  libpostproc    54.  0.100 / 54.  0.100
[NULL @ 0000000002488620] non-existing SPS 0 referenced in buffering period
[NULL @ 0000000002488620] SPS unavailable in decode_picture_timing
[h264 @ 000000000248a2a0] non-existing SPS 0 referenced in buffering period
[h264 @ 000000000248a2a0] SPS unavailable in decode_picture_timing
Input #0, hls,applehttp, from 'https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/encryption/bad/bad.m3u8':
  Duration: 00:00:29.95, start: 1.407333, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 23.98 fps, 23.98 tbr, 90k tbn, 47.95 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:bad-bad.mp4.part':
  Metadata:
    encoder         : Lavf57.52.100
    Stream #0:0: Video: h264 (Main) ([33][0][0][0] / 0x0021), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 23.98 fps, 23.98 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (LC) ([64][0][0][0] / 0x0040), 48000 Hz, stereo
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame=  270 fps=3.7 q=-1.0 Lsize=    4835kB time=00:00:22.50 bitrate=1759.8kbits/s speed=0.308x
video:4656kB audio:173kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.110669%
[ffmpeg] Downloaded 4950862 bytes
[download] 100% of 4.72MiB
...
<end of log>

m3u8 enc https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/encryption/bad/bad.m3u8
unencrypted video : https://s3.amazonaws.com/1d4326f61a9a4ed596de9e1a41d48413/input.ts

Description of your issue, suggested solution and other information

Can you add support for encrypted single file hls (m3u8) in the native hls extractor, since ffmpeg seem have bug reading the byte range resulted in corrupted / skipped video

https://tools.ietf.org/html/draft-pantos-http-live-streaming-20#section-4.3.2.2
https://developer.apple.com/library/content/technotes/tn2288/_index.html#//apple_ref/doc/uid/DTS40012238-CH1-BYTE_RANGE_
https://trac.ffmpeg.org/ticket/5858

also please add support for local m3u8 like youtube-dl.exe "C:\file.m3u8"
since sometime it's much faster downloading the encrypted file locally first (to avoid dropped frames)

thanks

yan12125 · 2016-10-18T09:21:46Z

That's already supported. Install python-crypto and run again.

TehBotolSosro · 2016-10-18T10:38:07Z

hi thanks for replying, but it still corrupt the video and in other site that use similar technique youtube-dl say "WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg" even after installing python-crypto in windows and add them to PATH

did i do something wrong or youtube-dl is also needed to be added to PATH?

other site m3u8

youtube-dl --hls-prefer-native http://
[generic] hlsc_whe2931_7: Requesting header
[generic] hlsc_whe2931_7: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: hlsc_whe2931_7-hlsc_whe2931_7.mp4
ffmpeg version N-82003-g9082603 Copyright (c) 2000-2016 the FFmpeg developers
built with gcc 5.4.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-libebur128 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
libavutil 55. 32.100 / 55. 32.100
libavcodec 57. 61.103 / 57. 61.103
libavformat 57. 52.100 / 57. 52.100
libavdevice 57. 0.102 / 57. 0.102
libavfilter 6. 64.100 / 6. 64.100
libswscale 4. 1.100 / 4. 1.100
libswresample 2. 2.100 / 2. 2.100
libpostproc 54. 0.100 / 54. 0.100
Input #0, hls,applehttp, from 'http://':
Duration: 01:29:17.71, start: 0.000000, bitrate: 0 kb/s
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 24 fps, 24 tbr, 90k tbn, 48 tbc
Metadata:
variant_bitrate : 0
Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
Metadata:
variant_bitrate : 0
Output #0, mp4, to 'file:hlsc_whe2931_7-hlsc_whe2931_7.mp4.part':
Metadata:
encoder : Lavf57.52.100
Stream #0:0: Video: h264 (High) ([33][0][0][0] / 0x0021), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 24 fps, 24 tbr, 90k tbn, 90k tbc
Metadata:
variant_bitrate : 0
Stream #0:1: Audio: aac (LC) ([64][0][0][0] / 0x0040), 44100 Hz, stereo
Metadata:
variant_bitrate : 0
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[hls,applehttp @ 00000000022b2ac0] Failed to open segment of playlist 02.0kbits/s speed=2.57x
Last message repeated 1 times
[https @ 00000000022bbee0] HTTP error 416 Requested Range Not Satisfiable
[crypto @ 00000000022bbee0] Unable to open resource: http://vuclip-a.akamaihd.net/9d34f5174054fba5e979a7d16a84b065/vp63207/9d34_e2931_7.vp.mov.ts
[hls,applehttp @ 00000000022b2ac0] Failed to open segment of playlist 0
[crypto @ 00000000022bbee0] Unable to open resource: http://vuclip-a.akamaihd.net/9d34f5174054fba5e979a7d16a84b065/vp63207/9d34_e2931_7.vp.mov.ts
[hls,applehttp @ 00000000022b2ac0] Failed to open segment of playlist 0
[crypto @ 00000000022bbee0] Unable to open resource: http://vuclip-a.akamaihd.net/9d34f5174054fba5e979a7d16a84b065/vp63207/9d34_e2931_7.vp.mov.ts
[hls,applehttp @ 00000000022b2ac0] Failed to open segment of playlist 0
---TRUNCATED---

result only the unencrypted video being downloaded while the encrypted video are not downloaded
thanks

yan12125 changed the title ~~Support for AES encrypted hls single file in native hls extractor~~ Support for #EXT-X-BYTERANGE in native hls extractor Oct 18, 2016

yan12125 added the request label Oct 18, 2016

yan12125 mentioned this issue Oct 21, 2016

drm vods #10989

Closed

4 of 8 tasks complete

remitamine added a commit that referenced this issue Apr 13, 2017

[downloader/hls] add basic support for EXT-X-BYTERANGE tag(#10955)

f597463

ytdl-org / youtube-dl

Support for #EXT-X-BYTERANGE in native hls extractor #10955

Support for #EXT-X-BYTERANGE in native hls extractor #10955

TehBotolSosro commented Oct 18, 2016

yan12125 commented Oct 18, 2016

TehBotolSosro commented Oct 18, 2016 •

edited

ytdl-org / youtube-dl

Join GitHub today

GitHub is where the world builds software

Support for #EXT-X-BYTERANGE in native hls extractor #10955

Support for #EXT-X-BYTERANGE in native hls extractor #10955

Comments

TehBotolSosro commented Oct 18, 2016

Please follow the guide below

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.10.16. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Description of your issue, suggested solution and other information

yan12125 commented Oct 18, 2016

TehBotolSosro commented Oct 18, 2016 • edited

Essential cookies

Always active

Analytics cookies

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2016.10.16. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

TehBotolSosro commented Oct 18, 2016 •

edited