[downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader #8201

remitamine · 2016-01-10T19:23:05Z

No description provided.

yan12125 · 2016-01-10T19:54:46Z

Does it provide the same functionality as #5882?

remitamine · 2016-01-10T20:04:52Z

Does it provide the same functionality as #5882?

i didn't test the branch in the other PR. but from the code it looks like it has the same functionality(except that i didn't change or use aes_cbc_decrypt because the current implementation is too slow when i try to use it the decryption time is more than the download time).

yan12125 · 2016-01-17T16:17:44Z

A side note: PyCrypto + Python 3.5.1 works for the video in #8261 but PyCrypto + Python 2.7.11 produces unplayable files.

remitamine · 2016-01-21T16:05:35Z

it works for me using PyCrypto + Python 2.7.11(i can play the video there is only a warning from ffmpeg):

python2 __main__.py -v -f hls-796 http://www.atresplayer.com/television/programas/equipo-de-investigacion/temporada-1/capitulo-148-coches-robados_2016011500469.html
[debug] System config: []
[debug] User config: [u'--external-downloader', u'aria2c', u'--external-downloader-args', u'-x 16', u'-f', u'[height<=?720]', u'--sub-lang', u'ar,arME,en,enUS,fr', u'--write-sub', u'--no-check-certificate']
[debug] Command-line args: [u'-v', u'-f', u'hls-796', u'http://www.atresplayer.com/television/programas/equipo-de-investigacion/temporada-1/capitulo-148-coches-robados_2016011500469.html']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.15
[debug] Git HEAD: 6e98ff7
[debug] Python version 2.7.11 - Linux-4.3.3-2-ARCH-x86_64-with-glibc2.2.5
[debug] exe versions: ffmpeg 2.8.4, ffprobe 2.8.4, rtmpdump 2.4
[debug] Proxy map: {}
[AtresPlayer] capitulo-148-coches-robados: Downloading webpage
[AtresPlayer] 20160115-EPISODE-00016-false: Downloading player JSON
[AtresPlayer] 20160115-EPISODE-00016-false: Downloading m3u8 information
[AtresPlayer] capitulo-148-coches-robados: Downloading timestamp
[AtresPlayer] capitulo-148-coches-robados: Downloading windows video JSON
[AtresPlayer] capitulo-148-coches-robados: Downloading f4m manifest
[AtresPlayer] capitulo-148-coches-robados: Downloading episode XML
[debug] Invoking downloader on u'http://deswowa3player.antena3.com/vsg/_definst_/smil:assets11/2016/01/15/6156CB72-2068-49A6-AB49-839D04FD429C/es.smil/chunklist_b724000.m3u8?pulse=assets11%2F2016%2F01%2F15%2F6156CB72-2068-49A6-AB49-839D04FD429C%2F%7C1453399869%7C51aae25dc31b8ba446e407df8f0b94d4'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 348
[download] Destination: Coches robados-capitulo-148-coches-robados.mp4
[download] 100% of 247.35MiB in 47:54

ffprobe Coches\ robados-capitulo-148-coches-robados.mp4 
ffprobe version 2.8.4 Copyright (c) 2007-2015 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[NULL @ 0x55ccdd32eac0] start time for stream 0 is not set in estimate_timings_from_pts
Input #0, mpegts, from 'Coches robados-capitulo-148-coches-robados.mp4':
  Duration: 00:57:53.80, start: 0.000000, bitrate: 597 kb/s
  Program 1 
    Stream #0:0[0x102]: Data: timed_id3 (ID3  / 0x20334449)
    Stream #0:1[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 640x360 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:2[0x101]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 50 kb/s
Unsupported codec with id 1414087731 for input stream 0

mpv Coches\ robados-capitulo-148-coches-robados.mp4 
Playing: Coches robados-capitulo-148-coches-robados.mp4
[ffmpeg] NULL: start time for stream 0 is not set in estimate_timings_from_pts
 (+) Video --vid=1 (h264)
 (+) Audio --aid=1 (aac)
AO: [pulse] 48000Hz stereo 2ch float
Using hardware decoding (vdpau).
VO: [vdpau] 640x360 vdpau
[vo/vdpau] Compositing window manager detected. Assuming timing info is inaccurate.
[vo/vdpau/x11] Disabling screensaver failed (4). Make sure the xdg-screensaver script is installed.
AV: 00:57:53 / 00:57:53 (99%) A-V:  0.000

yan12125 · 2016-01-21T16:26:13Z

It works now. I have recently switched between ffmpeg of my build and Arch Linux's official packages. Maybe there's something wrong in my build.

yan12125 · 2016-06-01T18:56:32Z

Could you rebase this PR to master? I'm testing against #9578, but ffmpeg does not work with proxies correctly.

remitamine · 2016-06-01T21:04:39Z

Could you rebase this PR to master?

done.

yan12125 · 2016-06-02T01:41:51Z

youtube_dl/downloader/hls.py

+                    frag_content = down.read()
+                    down.close()
+                    if decrypt_info['METHOD'] == 'AES-128':
+                        iv = decrypt_info.get('IV') or struct.pack(">8xq", media_sequence)


Use struct_pack.

remitamine · 2016-06-02T05:55:45Z

Use struct_pack.
Maybe it's better to some helper function in utils.py for parsing M3U8 manifests.

done.

yan12125 · 2016-06-11T04:29:15Z

youtube_dl/downloader/hls.py

+                        success = ctx['dl'].download(key_filename, {'url': decrypt_info['URI']})
+                        if not success:
+                            return False
+                        down, key_sanitized = sanitize_open(key_filename, 'rb')


I guess there's no need to download the key to the dist and read it from the disk. Just urlopen should be fine.

i did this as an optimization because lots of times the key is the same in the whole manifest with only the change of the IV(and don't download the key every time i resume the download even if the key is a small file) but it looks like this makes the downloading fail for videos that have different keys(found on ooyala).

Just urlopen should be fine.

i will use it for now as it will fix the problem with different keys.

…ive downloader

remitamine · 2016-06-19T00:06:45Z

Just urlopen should be fine.

done.

yan12125 · 2016-06-19T05:25:13Z

Thanks!

The pycrypto library is used for HLS streams with encrypted segments. See ytdl-org/youtube-dl#8201

* youtube-dl: add pycryptodome dependency The pycrypto library is used for HLS streams with encrypted segments. See ytdl-org/youtube-dl#8201 * youtube-dl: 2017.05.23 -> 2017.05.29

remitamine mentioned this pull request May 1, 2016

[downloader/hls] Delegate extraction to ffmpeg when unsupported features detected #9358

Merged

remitamine force-pushed the hls-aes branch 2 times, most recently from 3c20ec2 to 86e8845 Compare May 4, 2016 18:10

remitamine force-pushed the hls-aes branch from 86e8845 to 376ef26 Compare June 1, 2016 21:03

yan12125 reviewed Jun 2, 2016
View reviewed changes

remitamine force-pushed the hls-aes branch from 376ef26 to 4cbd342 Compare June 2, 2016 05:48

yan12125 reviewed Jun 11, 2016
View reviewed changes

yan12125 mentioned this pull request Jun 11, 2016

Drop Python 3.2 support #9751

Open

8 tasks

[downloader/hls] Add support for AES-128 encrypted segments in hlsnat…

e154c65

…ive downloader

remitamine force-pushed the hls-aes branch from 4cbd342 to e154c65 Compare June 19, 2016 00:05

yan12125 merged commit 6aeb64b into ytdl-org:master Jun 19, 2016

This was referenced Jun 19, 2016

Add support for encrypted media in NativeHlsFD #5882

Closed

Support AES encryption in native HLS downloader #6899

Closed

remitamine deleted the hls-aes branch June 29, 2016 16:48

yan12125 mentioned this pull request Sep 7, 2016

openload.co extractor not working #10408

Closed

asymmetric added a commit to asymmetric/nixpkgs that referenced this pull request Jun 2, 2017

youtube-dl: add pycryptodome dependency

4e1187f

The pycrypto library is used for HLS streams with encrypted segments. See ytdl-org/youtube-dl#8201

asymmetric mentioned this pull request Jun 2, 2017

youtube-dl: add pycryptodome dependency NixOS/nixpkgs#26329

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader #8201

[downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader #8201

remitamine commented Jan 10, 2016

yan12125 commented Jan 10, 2016

remitamine commented Jan 10, 2016

yan12125 commented Jan 17, 2016

remitamine commented Jan 21, 2016

yan12125 commented Jan 21, 2016

yan12125 commented Jun 1, 2016

remitamine commented Jun 1, 2016

yan12125 Jun 2, 2016

remitamine commented Jun 2, 2016

yan12125 Jun 11, 2016

remitamine Jun 18, 2016

remitamine commented Jun 19, 2016

yan12125 commented Jun 19, 2016

[downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader #8201

[downloader/hls] Add support for AES-128 encrypted segments in hlsnative downloader #8201

Conversation

remitamine commented Jan 10, 2016

yan12125 commented Jan 10, 2016

remitamine commented Jan 10, 2016

yan12125 commented Jan 17, 2016

remitamine commented Jan 21, 2016

yan12125 commented Jan 21, 2016

yan12125 commented Jun 1, 2016

remitamine commented Jun 1, 2016

yan12125 Jun 2, 2016

Choose a reason for hiding this comment

remitamine commented Jun 2, 2016

yan12125 Jun 11, 2016

Choose a reason for hiding this comment

remitamine Jun 18, 2016

Choose a reason for hiding this comment

remitamine commented Jun 19, 2016

yan12125 commented Jun 19, 2016