Twitch VOD download is not as fast as it could be

[Diazonium]

When I download a past broadcast from Twitch it is only using about 30% of my download speed, and the speed is very spiky. I believe this is due to that recent Twitch VODs seem to be stored in tiny chunks.(around 1MB) It would be nice if youtube-dl could get faster download speeds. This may be possible by downloading a number the chunks in parallel.

[jaimeMF]

We actually download big flv files, unfortunately they limit the bandwidth. You can workaround this by using a downloader like aria2c, for example youtube-dl 'http://www.twitch.tv/riotgames/b/579679336' --external-downloader aria2c (requires latest youtube-dl version) will download with a speed 2-3x higher.
There are already open issues for adding multithreaded download to youtube-dl, so I'm closing the issue.

For future reports, please post the full output of youtube-dl --verbose URL, because some errors are specific to the url and it allow us to detect common errors and outdated version.

[dstftw]

@jaimeMF, VODs are delivered differently rather than simple videos, namely it's HLS with m3u8 containing lots of small chunks.

[jaimeMF]

Sorry, I thought it was about about past broadcastasts. The method I posted doesn't apply then.

[phihag]

We should reuse the same TCP connection; that would probably give a great speedup.

[Diazonium]

Verbose output

youtube-dl --verbose http://www.twitch.tv/ara_gaming/v/3711528
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://www.twitch.tv/ara_gaming/v/3711528']
[debug] Encodings: locale cp1250, fs mbcs, out cp852, pref cp1250
[debug] youtube-dl version 2015.01.25
[debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-65738-g3c197442
[debug] Proxy map: {}
[twitch:vod] 3711528: Downloading vod info JSON
[twitch:vod] 3711528: Downloading vod access token
[twitch:vod] 3711528: Downloading m3u8 information
[debug] Invoking downloader on u'http://vod.ak.hls.ttvnw.net/v1/AUTH_system/vods_58ae/ara_
gaming_12760920816_193300649/chunked/index-dvr.m3u8'
[download] Destination: Hair of the dog is for crazy people. Tea please.-v3711528.mp4
ffmpeg version N-65738-g3c197442 Copyright (c) 2000-2014 the FFmpeg developers
built on Aug 18 2014 22:06:41 with gcc 4.8.3 (GCC)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --disable
-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-
gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca
--enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug
--enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libope
njpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable
-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc
--enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-l
ibwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-deckli
nk --enable-zlib
libavutil 54. 5.100 / 54. 5.100
libavcodec 56. 0.101 / 56. 0.101
libavformat 56. 1.100 / 56. 1.100
libavdevice 56. 0.100 / 56. 0.100
libavfilter 5. 0.100 / 5. 0.100
libswscale 3. 0.100 / 3. 0.100
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 0.100 / 53. 0.100
Input #0, hls,applehttp, from 'http://vod.ak.hls.ttvnw.net/v1/AUTH_system/vods_58ae/ara_ga
ming_12760920816_193300649/chunked/index-dvr.m3u8':
Duration: 02:58:33.71, start: 64.118000, bitrate: 0 kb/s
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 107 kb/s
Stream #0:1: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1920x108
0, 30 fps, 30 tbr, 90k tbn, 60 tbc
Stream #0:2: Data: timed_id3 (ID3 / 0x20334449)
Output #0, mp4, to 'Hair of the dog is for crazy people. Tea please.-v3711528.mp4.part':
Metadata:
encoder : Lavf56.1.100
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1920x1080, q=2-31, 30 fps,
90k tbn, 90k tbc
Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 107 kb/s
Stream mapping:
Stream #0:1 -> #0:0 (copy)
Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 3308 fps= 81 q=-1.0 Lsize= 42271kB time=00:01:50.74 bitrate=3126.9kbits/s
video:40394kB audio:1732kB subtitle:0kB other streams:0kB global headers:0kB muxing overhe
ad: 0.342595%
[ffmpeg] 43284996 bytes

[csenger]

This might be worth a separate issue additional to a general problem with slow http connections. But regarding @Diazonium's comment, there was a change in behavior on the twitch side.

Some Links to videos have the format http://www.twitch.tv/<username>/b/<videoid>, e.g. http://www.twitch.tv/riotgames/b/579679336 . youtube-dl downloads a bunch of flv files pretty fast.

csenger@sympathikus:~/Videos$ youtube-dl http://www.twitch.tv/riotgames/b/579679336
[twitch:video] 579679336: Downloading video info JSON
[twitch:video] 579679336: Downloading video playlist JSON
[download] Downloading playlist: 2014 Worlds Final - Samsung White vs. Star Horn Royal Club - 10/18/14
[twitch:video] playlist 2014 Worlds Final - Samsung White vs. Star Horn Royal Club - 10/18/14: Collected 13 video ids (downloading 13 of them)
[download] Downloading video 1 of 13
...

For other twitch users the URL format is different: http://www.twitch.tv/<username>/v/<videoid>, e.g. http://www.twitch.tv/planetside2/v/3810046 .
In this case youtube-dl tries to pipe the download through avconv. Maybe to combine the best audio with the best video, but I'm not sure. This produces the verbose output in the last comment and is pretty slow. Also aborted downloads can not be resumed.

[weedy]

[download] Downloading video 3 of 116
[generic] 67998156: Requesting header
[redirect] Following redirect to https://www.twitch.tv/moonmoon_ow/v/67998156
[twitch:vod] 67998156: Downloading vod info JSON
[twitch:vod] 67998156: Downloading vod access token
[twitch:vod] 67998156: Downloading m3u8 information
[download] Sleeping 3.0 seconds...
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 4515
[download] Destination: 20160522-v67998156-video_games.mp4
[download]  10.7% of ~3.70GiB at  523.27KiB/s ETA 01:21:38

If we know we are going to get a million chunks can we call --external-downloader with a list/file of all the chunks and use --external-downloader-args to parallel from there?

[yan12125]

If we know we are going to get a million chunks can we call --external-downloader with a list/file of all the chunks and use --external-downloader-args to parallel from there?

Such a feature is not implemented yet. See #6069

[lilydjwg]

I don't need to parallel those files. I prefer connection reuse which can normally speed up massive web page downloading by over 2x. For video chunks I haven't tested, but I can only run at about 200-500KiB/s, while downloading one large file gives me over 5MiB/s, from YouTube.

[yan12125]

@lilydjwg: Could you open a new issue for connection reuse? That's a different problem than the original one.

[lilydjwg]

@yan12125 done at #14523.

[rosshadden]

I know based on the above comments this isn't exactly what the OP was asking about, but I just wanted to say thank you to @jaimeMF for suggesting aria2c. It made my past broadcasts download at roughly 10-15x the speed as without.

[Gaploid]

@rosshadden How are you using aria2c? As I understand it does not support as an external downloader for HLS videos.

[rosshadden]

@Gaploid I passed --external-downloader aria2c. It worked great, and I have already watched and edited the resulting videos so I know it worked.

[Gaploid]

@rosshadden Ive tested a lot and it does not work for me. Is it [hlsnative] in logs when you are downloading?

[rosshadden]

I was talking about past broadcasts, not explicitly saved VODs. I believe the comments here went in the direction of VODs, which is what I meant when I said "this isn't exactly what the OP was asking about". Sorry for getting off topic; just wanted to say thanks. If we are actually on the same page and past broadcasts shouldn't be working or benefiting from aria2c, let me know and I'll do it again and look at logs.

[Gaploid]

Yep, I`m talking about past broadcasts. Maybe the low quality is saved in one file but 720p, 1080p is stored in hls format and youtube-dl does not support external downloaders.

[rosshadden]

@Gaploid Maybe that is the answer here. The video I fetched was https://go.twitch.tv/videos/183527029 which is very low quality. Twitch calls it 540p, though even that seems high.

youtube-dl --external-downloader aria2c https://go.twitch.tv/videos/183527029