parallel download

[bregon]

Is it possible to download from multiple url's at the same time?

[Hrxn]

Well, you can use aria2c to download, like this:

--external-downloader COMMAND    Use the specified external downloader.
                                 Currently supports
                                 aria2c,axel,curl,httpie,wget

[bregon]

If I have 1000 vimeo url's in a .txt file, can I download them 10 at a time?

The external downloaders seem to want to download one url from multiple sources or in multiple segments.

[Hrxn]

Uh, yes, youtube-dl hands only one URL at a time over to external downloaders like aria2c.

So, you can't do it this way. But you can always use Batchfiles (*.bat|*.cmd) (Shell scripts on Unixoids, etc.) to start youtube-dl. And you can run multiple batch files in parallel

[bregon]

That is what I have been doing until now. Very CPU intensive.

Would it be possible to add this feature to youtube-dl?

[Hrxn]

Very CPU intensive?

How many instances of youtube-dl were up and running?
You shouldn't use more than a couple, for example.

I don't really see the point. For sites like YouTube and Vimeo, for example, my download bandwidth is already saturated, you can't further 'accelerate' a download. I don't know about your internet connection, but I think it's highly unlikely that it works in any different way.

[bregon]

On my 2.3 GHz dual-core laptop I can only run about 20.

Is it possible to modify youtube-dl to handle multiple downloads?

[yan12125]

Duplicate of #3746.

[Hrxn]

Is it possible to modify youtube-dl to handle multiple downloads?

Yes, theoretically. But it's not feasible, it would require to rewrite significant parts of the program.
You can download in parallel when running youtube-dl in parallel.

But this doesn't help against sites that limit download rates somewhat, keep that in mind..

[alexanderluna]

My bandwidth is bigger than the download speed I get from most sites. When I batch download using the -a flag it caps at around 1mb/s. Using xargs I was able to use my full bandwidth:

xargs -n 1 -P 10 youtube-dl < my_video_links.txt

My bandwidth caps at around 10mb/s so this downloads 10 videos at the same time each caping at around 1-1.2mb/s. Hope this is helpful for someone