Please add support for multiple concurrent requests

[f4nff]

Now curl supports segmented requests through --range, but all this must be manual.
I can use

curl -D "dump1.txt" -H "Range: bytes=0-2000" https://www.baidu.com/img/bd_logo1.png -o part1
curl -D "dump2.txt" -H "Range: bytes=2001-4000" https://www.baidu.com/img/bd_logo1.png -o part2
curl -D "dump3.txt" -H "Range: bytes=4001-" https://www.baidu.com/img/bd_logo1.png -o part3

Segment request
then,
cat part1 part2 part3 >> logo.png
Merge, so that I can use three tcp connections to download files at the same time, and strive for greater bandwidth,
It would be great if these could all be automatic,

Such as

curl -X 3 https://www.baidu.com/img/bd_logo1.png -o logo.png
It will be very happy

[bagder]

Thanks, but this description sounds as if you're asking for a new feature/change. We use this tracker for bugs and issues only, we put ideas to work on in the future in the TODO document. We basically drown in good ideas so they don't do much use in our tracker.

If you really want to see this happen, start working on an implementation and submit a PR for it or join the mailing list and talk up more interest for it and see what help from others you can get!

[f4nff]

I can't find where the email is, because I see that curl supports range segmentation requests, and I can manually request multiple concurrent requests for a file, and then merge it, so that we can get higher through more connections The transmission rate is good.
On this basis, it seems not difficult to implement multiple concurrent requests. I very much hope to see the realization of this feature.

[bagder]

it seems not difficult to implement multiple concurrent requests

I agree. I don't think it is very difficult. My ideal vision of an implementation would also not use a specified number of parallel transfers, but curl could:

• Get the full file as transfer A
• If after N seconds have passed and the transfer is expected to continue for >M seconds more, add a new transfer (B) that asks for the second half of A's content (and stop A at the middle).
• if splitting up the work seemed to improve the transfer rate, it could then be done again. Then again, etc up to a limit.

This way, if transfer B fails (because Range: isn't supported) it will let transfer A remain the single one.

N and M could be set to some sensible defaults.

Still: this is not in my short term plan to work on personally.

[f4nff]

Your ideas seem to be more advanced, and more friendly and smart.
In the process of Internet transmission, one thing is very practical,
That is, if you download multiple files concurrently, you can get more bandwidth.
Many service providers limit the speed of a single connection. If there are multiple connections, it will lead to shorter and more efficient transmissions.
I look forward to your perfecting it.

[f4nff]

I am using aria2 to download more concurrently recently, but if the bandwidth exceeds 200Mbps, it becomes unstable.
Curl single concurrency is very stable. The only thing is that it does not support multiple concurrency. I look forward to supporting multiple concurrency.

[thenoobtester]

Now that curl has parallel download support. Is it possible to add different byte range headers per concurrent download in the config file?

For example if i put this in my config.txt config file:

 
    url = "http://www.example.com"
    header = "Range: bytes=0-1024"
    output = "chunk1"
    url = "http://www.example.com"
    header = "Range: bytes=1024-2048"
    output = "chunk2"
    url = "http://www.example.com"
    header = "Range: bytes=2048-3072"
    output = "chunk3"

and run

curl --parallel --parallel-max 60 --config config.txt

I get 3 chunks downloaded concurrently but always the first byte range 0-1024 instead of each specifc byte range per file. Is it impossible or is my config file incorrect?

Thank you for your help.

[dfandrich]

You probably need the "next" option between requests to make them use separate headers.

[thenoobtester]

@dfandrich You are my hero :) I put --next between each request in the config file and it worked!!!

[msoxzw]

it seems not difficult to implement multiple concurrent requests

I agree. I don't think it is very difficult. My ideal vision of an implementation would also not use a specified number of parallel transfers, but curl could:

* Get the full file as transfer A

* If after N seconds have passed and the transfer is expected to continue for >M seconds more, add a new transfer (B) that asks for the second half of A's content (and stop A at the middle).

* if splitting up the work seemed to improve the transfer rate, it could then be done again. Then again, etc up to a limit.

This way, if transfer B fails (because Range: isn't supported) it will let transfer A remain the single one.

N and M could be set to some sensible defaults.

Still: this is not in my short term plan to work on personally.

However, I think it is challenging if storage random access performance is bottleneck, e.g. hard disk or external storage. Regular single connection transfer guarantees sequential writing, but it is not the case for parallel transfer. The aria2 has many options to tune how data is written into storage, but I could not find an optimal solution, or aria2 cannot provide satisfactory balance among storage performance, re-connection overhead, and memory usage:

piece selection algorithm
memory-mapped file
file allocation

[AndriusCTR]

You probably need the "next" option between requests to make them use separate headers.

But then it won't work in parallel..?

"If you’re doing parallel transfers (with -Z), curl will do URLs in parallel only until the --next separator." - https://daniel.haxx.se/blog/2020/03/02/curl-ootw-next/

[hbednar]

I think what @AndriusCTR is meaning is segmented downloads, the below is a good description of it and how its different from something like multithreading

https://whatbox.ca/wiki/Multi-threaded_and_Segmented_FTP

[AndriusCTR]

I think what @AndriusCTR is meaning is segmented downloads, the below is a good description of it and how its different from something like multithreading

https://whatbox.ca/wiki/Multi-threaded_and_Segmented_FTP

Well no, I don't mean segmented downloads. I mean parallel downloads: get 10 URLs at once, but those URLs require 10 different cookies (for request headers).