Unable to download HEVC content from developer.apple.com

[greenywd]

Checklist

• I'm reporting a site feature request
• I've verified that I'm running youtube-dl version 2019.08.02
• I've searched the bugtracker for similar site feature requests including closed ones

Description

Hi!

Recently, Apple added hvc1 variants of their WWDC talks (plus 4K) for this year as seen from the output below:

└ youtube-dl -F https://developer.apple.com/videos/play/wwdc2019/231/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'https://developer.apple.com/videos/play/wwdc2019/231/', u'-v']
[debug] Encodings: locale US-ASCII, fs utf-8, out US-ASCII, pref US-ASCII
[debug] youtube-dl version 2019.08.02
[debug] Python version 2.7.16 (CPython) - Darwin-19.0.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 231: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 231: Downloading webpage
[generic] 231: Extracting information
[generic] 231: Downloading m3u8 information
[download] Downloading playlist: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer
[generic] playlist Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[info] Available formats for 231:
format code                          extension  resolution note
hls-program_audio-Alternate_Audio    mp4        audio only [eng]
hls-program_audio_0-Alternate_Audio  mp4        audio only [eng]
hls-542                              mp4        480x270     542k , avc1.640015, 29.97fps, video only
hls-747                              mp4        640x360     747k , avc1.64001f, 29.97fps, video only
hls-1276                             mp4        640x360    1276k , hvc1.2.4.H150.B0, 29.97fps, video only
hls-1559                             mp4        960x540    1559k , hvc1.2.4.H150.B0, 29.97fps, video only
hls-3049                             mp4        1280x720   3049k , hvc1.2.4.H150.B0, 29.97fps, video only
hls-4514                             mp4        1920x1080  4514k , hvc1.2.4.H150.B0, 29.97fps, video only
hls-4630                             mp4        960x540    4630k , avc1.64001f, 29.97fps, video only
hls-6266                             mp4        1280x720   6266k , avc1.64001f, 29.97fps, video only
hls-8821                             mp4        1920x1080  8821k , avc1.640028, 29.97fps, video only
hls-11670                            mp4        3840x2160  11670k , hvc1.2.4.H150.B0, 29.97fps, video only
hls-16465                            mp4        3840x2160  16465k , hvc1.2.4.H150.B0, 29.97fps, video only (best)
[download] Finished downloading playlist: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer

Downloading the avc1 streams work perfectly - just the newer hvc1 ones don't.

Both youtube-dl and ffmpeg can't properly download these - unclear of the reasoning. Using youtube-dl -f hls-1276 https://developer.apple.com/videos/play/wwdc2019/231/ appears to use ffmpeg, which spits out a few hundred warnings over a few seconds, then appears to open the same URL for the duration of the "download". An output can be found here - https://pastebin.com/dbXaCebw (apologies, but it was waaaay to big to paste here).

The final download is around a 10 second video, and not the full length of the session. Below is a mediainfo in case it helps.

Mediainfo

└ mediainfo Integrating\ SwiftUI\ -\ WWDC\ 2019\ -\ Videos\ -\ Apple\ Developer-231.mp4
General
Complete name                            : Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer-231.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/mp41)
File size                                : 4.12 MiB
Duration                                 : 11 s 493 ms
Overall bit rate                         : 3 005 kb/s
Writing application                      : Lavf58.20.100

Video
ID                                       : 1
Format                                   : HEVC
Format/Info                              : High Efficiency Video Coding
Format profile                           : Main 10@L5@High
Codec ID                                 : hev1
Codec ID/Info                            : High Efficiency Video Coding
Duration                                 : 11 s 493 ms
Bit rate                                 : 3 002 kb/s
Width                                    : 640 pixels
Height                                   : 360 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Variable
Frame rate                               : 67.869 FPS
Minimum frame rate                       : 1.765 FPS
Maximum frame rate                       : 30 000.000 FPS
Original frame rate                      : 29.970 (30000/1001) FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 10 bits
Bits/(Pixel*Frame)                       : 0.192
Stream size                              : 4.11 MiB (100%)
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709
Codec configuration box                  : hvcC

I read through the FAQ and found that we can choose to prefer native HLS over ffmpeg, which resulted in a file that appeared to have been downloaded correctly (as it was larger than a few mb), but no video players could open it. Again, here's the output of youtube-dl and the mediainfo of the file it downloaded.

youtube-dl

└ youtube-dl -f hls-1276 --hls-prefer-native https://developer.apple.com/videos/play/wwdc2019/231/ -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-f', u'hls-1276', u'--hls-prefer-native', u'https://developer.apple.com/videos/play/wwdc2019/231/', u'-v']
[debug] Encodings: locale US-ASCII, fs utf-8, out US-ASCII, pref US-ASCII
[debug] youtube-dl version 2019.08.02
[debug] Python version 2.7.16 (CPython) - Darwin-19.0.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 231: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 231: Downloading webpage
[generic] 231: Extracting information
[generic] 231: Downloading m3u8 information
[download] Downloading playlist: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer
[generic] playlist Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[debug] Invoking downloader on u'https://devstreaming-cdn.apple.com/videos/wwdc/2019/231qbm6xl2bbd5t/231/0640c/prog_index.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 383
[download] Destination: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer-231.mp4
[download] 100% of 235.65MiB in 05:07
[debug] ffmpeg command line: ffprobe -show_streams 'file:Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer-231.mp4'
[download] Finished downloading playlist: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer

Mediainfo

└ mediainfo Integrating\ SwiftUI\ -\ WWDC\ 2019\ -\ Videos\ -\ Apple\ Developer-231.mp4
General
Complete name                            : Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer-231.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 236 MiB

Strangely enough, I was unable to play the playlist (m3u8) in any other browser than Safari on macOS (unable to test on Linux/Windows atm). VLC can also play these on any platform perfectly by giving it the m3u8 of a HEVC stream (i.e. https://devstreaming-cdn.apple.com/videos/wwdc/2019/231qbm6xl2bbd5t/231/0640c/prog_index.m3u8).

[dstftw]

> py -3.7 .\youtube_dl\__main__.py https://developer.apple.com/videos/play/wwdc2019/231/  --no-chec -f hls-1276
[generic] 231: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 231: Downloading webpage
[generic] 231: Extracting information
[generic] 231: Downloading m3u8 information
[download] Downloading playlist: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer
[generic] playlist Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer: Collected 1 video ids (downloading 1 of them)
[download] Downloading video 1 of 1
[download] Destination: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer-231.mp4
ffmpeg version N-85653-gb4330a0 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 6.3.0 (GCC)
  configuration: --enable-gpl --enable-version3 --enable-cuda --enable-cuvid --enable-d3d11va --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-zlib
  libavutil      55. 61.100 / 55. 61.100
  libavcodec     57. 93.100 / 57. 93.100
  libavformat    57. 72.101 / 57. 72.101
  libavdevice    57.  7.100 / 57.  7.100
  libavfilter     6. 87.100 /  6. 87.100
  libswscale      4.  7.101 /  4.  7.101
  libswresample   2.  8.100 /  2.  8.100
  libpostproc    54.  6.100 / 54.  6.100
Input #0, hls,applehttp, from 'https://devstreaming-cdn.apple.com/videos/wwdc/2019/231qbm6xl2bbd5t/231/0640c/prog_index.m3u8':
  Duration: 00:38:21.93, start: 10.033367, bitrate: 0 kb/s
  Program 0
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: hevc (Main 10) (hev1 / 0x31766568), yuv420p10le(tv, bt709), 640x360 [SAR 1:1 DAR 16:9], 280 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 29.97 tbc
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer-231.mp4.part':
  Metadata:
    encoder         : Lavf57.72.101
    Stream #0:0: Video: hevc (Main 10) ([35][0][0][0] / 0x0023), yuv420p10le(tv, bt709), 640x360 [SAR 1:1 DAR 16:9], q=2-31, 280 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame=68989 fps=285 q=-1.0 Lsize=  240995kB time=00:38:21.86 bitrate= 857.7kbits/s speed=9.51x
video:240349kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.268958%
[ffmpeg] Downloaded 246779283 bytes
[download] 100% of 235.35MiB in 04:03
[download] Finished downloading playlist: Integrating SwiftUI - WWDC 2019 - Videos - Apple Developer

[greenywd]

@dstftw hmm, what OS are you running? I tried on Arch and get the exact same results as on macOS - even when using the same YouTube-dl flags as you. I'll compare our ffmpeg configurations just in case and try on both Windows and a newer MacBook later.

Edit: Windows 10 has the same issue
Double Edit: Looks like a newer ffmpeg broke/dropped support for it. Swapped to your version and it downloads fine. Thanks for the help!