Ranged PUT request with CURLOPT_RESUME_FROM does not work

[simon-berger]

I did this

I wanted to do a ranged put request as described here.

// Setup curl
std::shared_ptr<CurlHolder> curl_(new CurlHolder());

// Set request url
std::string url{"http://httpbin.org/put"};
curl_easy_setopt(curl_->handle, CURLOPT_URL, url.c_str());

// Set payload
std::string body{"world"};
curl_easy_setopt(curl_->handle, CURLOPT_POSTFIELDSIZE_LARGE, static_cast<curl_off_t>(body.length()));
curl_easy_setopt(curl_->handle, CURLOPT_COPYPOSTFIELDS, body.c_str());

// Upload byte 6 to 10 of string "hello world" -> "world" (5 bytes) using a ranged PUT request
// Range values
curl_off_t resume_from = static_cast<curl_off_t>(6);
curl_off_t filesize = static_cast<curl_off_t>(11);

// Create custom Content-Range header described in https://curl.se/mail/lib-2019-05/0012.html
constexpr int max_header_char_count{256};
std::array<char, max_header_char_count> range_header{};
sprintf(range_header.data(),
        "Content-Range: bytes %" CURL_FORMAT_CURL_OFF_T
        "-"
        "%" CURL_FORMAT_CURL_OFF_T "/*",                      /**/
        resume_from, filesize - 1);
std::string range_header_str = std::string(range_header.data());
curl_slist* chunk = nullptr;
chunk = curl_slist_append(chunk, range_header_str.c_str());
curl_easy_setopt(curl_->handle, CURLOPT_HTTPHEADER, chunk);
curl_slist_free_all(curl_->chunk);
curl_->chunk = chunk;

// Set range values
curl_easy_setopt(curl_->handle, CURLOPT_RESUME_FROM_LARGE, resume_from);
curl_easy_setopt(curl_->handle, CURLOPT_INFILESIZE_LARGE, filesize);

// Perform PUT request
curl_easy_setopt(curl_->handle, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_perform(curl_->handle);

I expected the following

Execute a PUT request to the given server url with an upload range.

Actual behavior

No HTTP request is executed (checked with Wireshark) and instead, the execution of the program is stuck at curl_easy_perform(curl_->handle). The request is only executed if I comment / remove curl_easy_setopt(curl_->handle, CURLOPT_RESUME_FROM_LARGE, resume_from which results in disabling the resume from option.
Note: The request is also executed if I remove the body.

What I have tried to fix the problem

• Don't set custom Content-Range header
• Different values for resume_from and filesize
• Some more details about the issue can be found here: Add support for uploading arbitrary file parts with CURLOPT_RESUME_FROM libcpr/cpr#753
  • Here I have also outlined why I think that the following code might be interesting:
    https://github.com/curl/curl/blob/c9b60f005358a364cbcddbebd8d12593acffdd84/lib/http.c#L2856-2908

curl/libcurl version

curl-7.83.1
Same problem with curl-7.81.0 which I have used before i encountered the problem.

operating system

Ubuntu 20.04.4 LTS
Linux Simon-PC 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[bagder]

HTTP does not support range "uploads" with PUT or POST.

You can do an implementation, but then you need a server side to agree with the specific method you decide to use from the client and it will not be a generic solution with any HTTP client/server combination.

[simon-berger]

Okay, I understand that.
However, shouldn't curl at least be able to send the request to the server?

Otherwise, that should probably be adjusted in the docs:

For HTTP PUT uploads this option should not be used, since it may conflict with other options. If you need to upload arbitrary parts of a file (like for Amazon's web services) support is limited. We suggest set resume position using CURLOPT_RESUME_FROM, set end (resume+size) position using CURLOPT_INFILESIZE and seek to the resume position before initiating the transfer for each part. For more information refer to https://curl.se/mail/lib-2019-05/0012.html

[bagder]

What's the request you want it to send, exactly? Since you're asking for a feature HTTP does not provide it isn't clear to me.

[simon-berger]

I want to send a request like it is described here.

For example say your input file size is 6 and your part size is 4 then
there would be two transfers.
Content-Range: bytes 0-3/*
Content-Length: 4
Content-Range: bytes 4-5/*
Content-Length: 2

[bagder]

To upload 4 bytes with a custom Content-Range: header:

$ ./src/curl -d data -H "Content-Range: bytes 0-3/*" -v localhost

It creates this request:

POST / HTTP/1.1
Host: localhost
User-Agent: curl/7.84.0-DEV
Accept: */*
Content-Range: bytes 0-3/*
Content-Length: 4
Content-Type: application/x-www-form-urlencoded

...

[simon-berger]

Ok, many thanks for the answer 👍
So I wrongly assumed that one can use the CURLOPT_RESUME_FROM option for PUT uploads.
Then I think it would be better to remove the section from the CURLOPT_RANGE documentation which describes this to avoid misunderstandings in the future.

[jay]

You have to use CURLOPT_UPLOAD as discussed in the previous e-mail. CURLOPT_CUSTOMREQUEST, "PUT" will not do what you expect:

When you change the request method by setting CURLOPT_CUSTOMREQUEST to something, you do not actually change how libcurl behaves or acts in regards to the particular request method, it will only change the actual string sent in the request.

Regardless maybe I was wrong in the e-mail. You'd have to have a custom read function to stop reading, I think, rather than rely on the default read function in libcurl which will keep reading past the part you want to send.

[bagder]

With @jay's updated man page landed, I think we can close this issue.