Multi-Threaded downloading for playlists

[JacobHacker]

When downloading playlists youtube-dl could use multiple connections to download videos at the same time, right now my dl speed averages at 50k/s, and the full playlist is about 15G (defcon19).

I've been running 4 terminals with --playlist-start=N so far, and spaced them out.

[phil2508]

This technique is used by desktop apps and is very useful in increasing download speed. I agree that 50k/s is way too slow.

[reicab]

I agree that this is essential - it is the only issue I have with the youtube-dl library but its a big issue. multi-threaded dowloading and the resulting increase in speed is a dire necessity.

[joshua-hull]

Well there might be other reasons you're only getting 50k/s. It could be an issue with your ISP or the site. All that being said I too would love to see concurrent or multi threaded downloads.

Sent via Hubroid

[phil2508]

It's actually Youtube slowing it down after the first 30% have been cached. I'm running this on a box with dedicated 200MB/s internet connection.

[konisto]

Yes, it is youtube and yes, PLEASE add multithreaded downloading. we promote a desktop app downloading youtube apps and multithreading is crucial for use experience.

we need to be able to offer the same speed for online conversions... thanks

[phil2508]

any chance this will be going somewhere? Are we the only ones annoyed by youtube throttling our downloads? thanks!

[dev0k]

Hi!

use external downloaders (like wget, axel) and get full speed. :)

[phil2508]

Wonderful idea. I've just implemented Axel and get 15MB/s now instead of 200k/s. Very nice indeed. Thanks so much for pointing me this way.

The only thing I don't like is that did I not think of this myself??!! Seems so easy.

[phil2508]

By the way, wget does not make sense as it is single threaded, too and gets the same throttling as youtube-dl.py

[singhpratyush]

I tried many options and here is the best that I came up with - use aria2.

The --external-downloader option

$ youtube-dl --external-downloader aria2c --external-downloader-args '-x <number of threads>' https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6

But this will download only one file at a time. To download many videos simultaneously, you will have to prepopulate download links.

Prepopulate download links

$ youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > dlinks.txt
$ aria2c -i dlinks.txt -j <number of simultaneous downloads> -x <number of threads for each download>

You will have to use >> instead of > in the first command to append more links.

The two commands may also be combined -

$ aria2c -j <number of simultaneous downloads> -x <number of threads for each download> `youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6`

But this will result in no backup of your links in case download interrupts.

Note: In case of prepopulating links, you may end up in a state when your links get expired because your previous downloads would have taken a while.

[yan12125]

Let me write some summary. Here are several levels to implement threaded downloading in youtube-dl:

1. User-input URLs, including command line options and batch files. (#109)
2. Playlists (this issue)
3. Segments in segmented streams (HLS, HDS, DASH) (#11044)
4. Chunk level in a single URL (#1498)

Only the last case works with external downloaders like aria2c or axel. For the first three kinds, more codes are necessary.

[d3fault]

Multi-Threaded postproc for playlists

Was tempted to open another issue for it, but this is too similar (edit: as suggested, I put this in a separate issue #12215). I'm fine with single threaded downloads (multi threaded downloads introduce the possibility of getting your IP banned), but I'd like to see the postproc done in separate thread(s) (up to N threads, N being the number of CPU cores). As soon as a download is finished, the postproc job should be dispatched and the next download should start immediately. But if all N cores are doing postproc, wait for one to finish before downloading another playlist item. If the machine has only 1 core then the the behavior is exactly the same as it already is.

On my setup postproc is the bottleneck, not bandwidth. Postproc benefits much more from mult-threading, whereas downloading only benefits from multi-threading if the server is throttling you (which implies they don't want you to use multiple connections simultaneously, and might ban you if you do).

[yan12125]

@d3fault: Thanks for writing this. Technically, postprocessing and downloading are implemented in different modules, it's better to have an separate issue.

[TyraelElDruin]

I wrote a multithreaded youtube-dl GUI. It basically just wraps the youtube-dl process and calls it for as many links as you set. The playlist portion does not run multithreaded however - before I wrapped youtube-dl I wrote the code to pull the links automatically, but since youtube-dl already handled it in a single thread, I left it alone. You could use it to download multiple playlists at a time though. One thing I've done for personal use is inject some javascript in the youtube playlist browser that just pulls the links into a text box at the bottom. You end up with a list of links and you can just paste them in and watch the multithreaded magic go. You can see it here https://www.youtube.com/watch?v=ihEH4DuTcTg

[ncovercash]

Will this ever be put in the main program?

[Gaploid]

Hey Guys, any plans to add multithreaded HLS downloading feature?

[crypdick]

@singhpratyush Is it possible to modify your command to extract audio? In particular, I want to extract audio-- I usually run youtube-dl --extract-audio --format bestaudio --external-downloader aria2c --external-downloader-args '--max-connection-per-server=15' --yes-playlist "<plist_url>"

[singhpratyush]

@crypdick: Yes, it would work fine. You can also do postprocessing on the audio if you want, e.g. --audio-format mp3.

[oltodosel]

my sh snippet

ytall() {
	special_o='%(upload_date)s__%(title)s-%(id)s.%(ext)s'
	default_threads=3
	
	if [[ -z $1 ]]; then
		echo "ytall [channel_url/playlist] [threads] [special -o ${special_o}]"
		return
	fi
	
	if [[ -n $3 ]]; then
		youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $2 --progress youtube-dl -i -o \'$special_o\' -- 
	elif [[ -n $2 ]]; then
		youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $2 --progress youtube-dl -i --
	else
		youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $default_threads --progress youtube-dl -i --
	fi
}

[rautamiekka]

@oltodosel that's a good snippet, but there are 2 things I notice right away:

1. You're limiting the max number of channels/playlists to 1 through the order of the parameters.

2. You're referencing parameter 1 without quoting it.

[oltodosel]

@rautamiekka

1. I don't need it.
2. You're right, guess I got too reliant on oh-my-zsh's automatic url quotation.

[MPeti1]

I tried many options and here is the best that I came up with - use aria2.

The --external-downloader option

$ youtube-dl --external-downloader aria2c --external-downloader-args '-x <number of threads>' https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6

But this will download only one file at a time. To download many videos simultaneously, you will have to prepopulate download links.

Prepopulate download links

$ youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > dlinks.txt
$ aria2c -i dlinks.txt -j <number of simultaneous downloads> -x <number of threads for each download>

You will have to use >> instead of > in the first command to append more links.

The two commands may also be combined -

$ aria2c -j <number of simultaneous downloads> -x <number of threads for each download> `youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6`

But this will result in no backup of your links in case download interrupts.

Note: In case of prepopulating links, you may end up in a state when your links get expired because your previous downloads would have taken a while.

There are playlists for which the -g option returns links that start with manifest.googlevideo.com, and if such a link is opened in a browser it will download a few hundred KB file. Is there a solution for cases like that?
Note: your playlist still returns the old links which aria2c can download