Adult swim: Improved downloading with "-re" to ffmpeg

[keybounce]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.06.05. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• [ x] I've verified and I assure that I'm running youtube-dl 2017.06.05

Before submitting an issue make sure you have:

• [x ] At least skimmed through README and most notably FAQ and BUGS sections
• [x ] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

• [x ] Bug report (encountered problems with youtube-dl)
• [x ] Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

So attempting to download videos from adult swim normally results in ffmpeg doing a download as fast as possible, and then the video freezing/dying in the middle.

Manually fetching, with the "--dump-json" and creating an ffmpeg command line, and adding "-re" to force real-time downloading works.

I'm not going to call it perfect -- I got this:

[hls,applehttp @ 0x7f8bc400ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment16.ts', offset 0, playlist 0
[hls,applehttp @ 0x7f8bc400ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment17.ts', offset 0, playlist 0
[hls,applehttp @ 0x7f8bc400ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment18.ts', offset 0, playlist 0
[hls,applehttp @ 0x7f8bc400ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment19.ts', offset 0, playlist 0
[crypto @ 0x7f8bc5800dc0] Unable to open resource: http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment19.ts
[hls,applehttp @ 0x7f8bc400ac00] Failed to open segment of playlist 0
[hls,applehttp @ 0x7f8bc400ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment20.ts', offset 0, playlist 0
[hls,applehttp @ 0x7f8bc400ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment21.ts', offset 0, playlist 0

(as I say this, the file is 110 segments and 18 minutes into the download).

Now, I cannot manage to get ffmpeg to resume an interrupted download no matter what I do (any error requires restarting from the beginning). My understanding of the native downloader inside youtube-dl is that it can resume. While getting most of the video is better than nothing, being able to retry would be even better.

NB: I have had horrible network performance tonight (fault on my router). It is entirely possible that that "unable to open" was entirely on my own end. So when I say I'd like to see a failure like that result in at least one retry/resume, I am serious.

===

The TL;DR: 1. Adding "-re" to ffmpeg options on adult swim turns a large number of failures into successes.
2. ffmpeg doesn't seem able to resume an interrupted download.
3. If the internal downloader can resume, then switching would permit restarting from network errors.

[keybounce]

GRR. Or not.

It died (stopped downloading) on segment 113.

Its working a whole lot better, but not perfect, and lacking a "Resume" just doesn't work.

Video being tested:
http://www.adultswim.com/videos/dragon-ball-z-kai/stunning-youngsters-no-18s-uphill-battle/

[keybounce]

Just an additional bit of info:
May 15th and 16th, no problem downloading a bunch of videos.
June 5th, downloaded these:

 575520 -rw-r--r--  1 michael  staff  589330725 Jun  5 21:56 Dragon Ball Z Kai/s01-e117-Dragon Ball Z Kai - Whither Victory A Battle Royal Championship Match!.mp4
 575324 -rw-r--r--  1 michael  staff  589129182 Jun  5 21:42 Dragon Ball Z Kai/s01-e116-Dragon Ball Z Kai - Don't Sell Super Saiyans Short! Vegeta and Goku's Full-Bore Power.mp4
 133112 -rw-r--r--  1 michael  staff  136302756 Jun  5 17:35 Sand Whale and Me/s01-e01-Sand Whale and Me - Day 1 - Warfare Encounter.mp4
 326680 -rw-r--r--  1 michael  staff  334520225 Jun  5 16:30 Tokyo Ghoul/s01-e02-Tokyo Ghoul - Tokyo Ghoul 02.mp4
 302800 -rw-r--r--  1 michael  staff  310064490 Jun  5 16:30 Dragon Ball Super/s01-e12-Dragon Ball Super - Dragon Ball Super 12.mp4
 355876 -rw-r--r--  1 michael  staff  364414306 Jun  5 16:26 Tokyo Ghoul/s01-e01-Tokyo Ghoul - Tokyo Ghoul 01.mp4
 319196 -rw-r--r--  2 michael  staff  326856409 Jun  5 16:25 Samurai Jack/s05-e10-Samurai Jack - Samurai Jack 101.mp4
 589792 -rw-r--r--  1 michael  staff  603943806 Jun  5 16:22 Samurai Jack/s05-e09-Samurai Jack - C.mp4
 579300 -rw-r--r--  1 michael  staff  593202730 Jun  5 16:13 Samurai Jack/s05-e08-Samurai Jack - XCIX.mp4

After that, problem. I can watch videos in the browser just fine. Downloading is failing.

[keybounce]

So attempting to use the rate-limit option, -r, does not affect ffmpeg. Neither does --keep-fragments.

--hls-prefer-native still uses ffmpeg.

And just in case, I even tried --include-ads, to see if that was triggering the "this is a browser, let it happen" test.

All fails.

keybounceMBP:AdultSwim michael$ ./yt-dl http://www.adultswim.com/videos/dragon-ball-z-kai/stunning-youngsters-no-18s-uphill-battle/
[debug] System config: []
[debug] User config: ['-k', '-o', '%(title)s-%(timestamp)6i.%(ext)s', '-f', '\nbest[ext=mp4][height>431][height<=576]/\nbestvideo[ext=mp4][height=480]+bestaudio[ext=m4a]/\nbest[ext=mp4][height>340][height<=431]/\nbestvideo[ext=mp4][height>360][height<=576]+bestaudio/\nbest[height>340][height<=576]/\nbestvideo[height>360][height<=576]+bestaudio/\nbestvideo[height=360]+bestaudio/\nbest[ext=mp4][height>=280][height<=360]/\nbest[height<=576]/\nworst', '--ap-mso', 'Dish', '--ap-username', 'PRIVATE', '--ap-password', 'PRIVATE', '--write-sub', '--write-auto-sub', '--sub-lang', 'en,enUS,en-us', '--sub-format', 'ass/srt/best', '--convert-subs', 'ass', '--embed-subs', '--recode-video', 'mp4', '--mark-watched', '--download-archive', 'downloaded-videos.txt']
[debug] Custom config: []
[debug] Command-line args: ['-v', '--hls-prefer-native', '--keep-fragments', '-r', '230k', '-o', '%(series)s/s%(season_number)02d-e%(episode_number)02d-%(title)s.%(ext)s', '--include-ads', 'http://www.adultswim.com/videos/dragon-ball-z-kai/stunning-youngsters-no-18s-uphill-battle/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.06.05
[debug] Python version 3.6.1 - Darwin-13.4.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 3.2.4, ffprobe 3.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[AdultSwim] stunning-youngsters-no-18s-uphill-battle: Downloading webpage
[AdultSwim] yg6tkv1KS3yUltXY4HS5zA: Downloading XML
[AdultSwim] 1400176: Retrieving Media Token
[AdultSwim] 1400176: Downloading XML
[AdultSwim] 1400176: Downloading m3u8 information
[AdultSwim] 1400176: Downloading m3u8 information
[debug] Invoking downloader on 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6_vod.m3u8'
[hlsnative] Downloading m3u8 manifest
WARNING: hlsnative has detected features it does not support, extraction will be delegated to ffmpeg
[download] Destination: Dragon Ball Z Kai/s01-e118-Dragon Ball Z Kai - DBZ Kai 118.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -seekable 0 -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
Cookie: hdnea2=expires=1496986306~access=/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/*~md5=bc9fb1e64b5098d457079391461e6ecb
' -i http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6_vod.m3u8 -c copy -f mp4 'file:Dragon Ball Z Kai/s01-e118-Dragon Ball Z Kai - DBZ Kai 118.mp4.part'
ffmpeg version 3.2.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 6.0 (clang-600.0.57) (based on LLVM 3.5svn)
  configuration: --prefix=/opt/local --enable-swscale --enable-avfilter --enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus --enable-libtheora --enable-libschroedinger --enable-libopenjpeg --enable-libmodplug --enable-libvpx --enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray --enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype --enable-libfribidi --disable-libxcb --disable-libxcb-shm --disable-libxcb-xfixes --disable-indev=jack --disable-outdev=xv --enable-audiotoolbox --enable-sdl2 --mandir=/opt/local/share/man --enable-shared --enable-pthreads --cc=/usr/bin/clang --enable-vda --enable-videotoolbox --arch=x86_64 --enable-yasm --enable-libx265 --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment0.ts', offset 0, playlist 0
[h264 @ 0x7fc3b301f000] Reinit context to 848x480, pix_fmt: yuv420p
Input #0, hls,applehttp, from 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6_vod.m3u8':
  Duration: 00:21:45.98, start: 0.750000, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Constrained Baseline), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(tv, left), 848x480, Closed Captions, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (HE-AAC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp
    Metadata:
      variant_bitrate : 0
Output #0, mp4, to 'file:Dragon Ball Z Kai/s01-e118-Dragon Ball Z Kai - DBZ Kai 118.mp4.part':
  Metadata:
    encoder         : Lavf57.56.101
    Stream #0:0: Video: h264 (Constrained Baseline), 1 reference frame ([33][0][0][0] / 0x0021), yuv420p(tv, left), 848x480 (0x0), q=2-31, 29.97 fps, 29.97 tbr, 90k tbn, 90k tbc
    Metadata:
      variant_bitrate : 0
    Stream #0:1: Audio: aac (HE-AAC) ([64][0][0][0] / 0x0040), 44100 Hz, stereo
    Metadata:
      variant_bitrate : 0
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
Automatically inserted bitstream filter 'aac_adtstoasc'; args=''
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment1.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment2.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment3.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment4.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment5.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment6.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment7.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment8.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment9.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment10.ts', offset 0, playlist 0
[hls,applehttp @ 0x7fc3b180ac00] HLS request for url 'http://androidhls-secure.cdn.turner.com/adultswim/big/secure/tve/2017-05/ASOE1004271700075224_HLS/ASOE1004271700075224_Layer6/8996_Period1/segment11.ts', offset 0, playlist 0

(locked up)

And just in case you are curious, I tried some other videos. All the same.

I have not had anything work since things stopped working on the 5th.

[dstftw]

Most likely network related issue.

--external-downloader-args to pass arguments to ffmpeg.

--hls-prefer-native still uses ffmpeg.

Install pycrypto.

[keybounce]

Thank you.