Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ranged PUT request with CURLOPT_RESUME_FROM does not work #8969

Closed
simon-berger opened this issue Jun 6, 2022 · 8 comments
Closed

Ranged PUT request with CURLOPT_RESUME_FROM does not work #8969

simon-berger opened this issue Jun 6, 2022 · 8 comments
Labels

Comments

@simon-berger
Copy link

@simon-berger simon-berger commented Jun 6, 2022

I did this

I wanted to do a ranged put request as described here.

// Setup curl
std::shared_ptr<CurlHolder> curl_(new CurlHolder());

// Set request url
std::string url{"http://httpbin.org/put"};
curl_easy_setopt(curl_->handle, CURLOPT_URL, url.c_str());

// Set payload
std::string body{"world"};
curl_easy_setopt(curl_->handle, CURLOPT_POSTFIELDSIZE_LARGE, static_cast<curl_off_t>(body.length()));
curl_easy_setopt(curl_->handle, CURLOPT_COPYPOSTFIELDS, body.c_str());

// Upload byte 6 to 10 of string "hello world" -> "world" (5 bytes) using a ranged PUT request
// Range values
curl_off_t resume_from = static_cast<curl_off_t>(6);
curl_off_t filesize = static_cast<curl_off_t>(11);

// Create custom Content-Range header described in https://curl.se/mail/lib-2019-05/0012.html
constexpr int max_header_char_count{256};
std::array<char, max_header_char_count> range_header{};
sprintf(range_header.data(),
        "Content-Range: bytes %" CURL_FORMAT_CURL_OFF_T
        "-"
        "%" CURL_FORMAT_CURL_OFF_T "/*",                      /**/
        resume_from, filesize - 1);
std::string range_header_str = std::string(range_header.data());
curl_slist* chunk = nullptr;
chunk = curl_slist_append(chunk, range_header_str.c_str());
curl_easy_setopt(curl_->handle, CURLOPT_HTTPHEADER, chunk);
curl_slist_free_all(curl_->chunk);
curl_->chunk = chunk;

// Set range values
curl_easy_setopt(curl_->handle, CURLOPT_RESUME_FROM_LARGE, resume_from);
curl_easy_setopt(curl_->handle, CURLOPT_INFILESIZE_LARGE, filesize);

// Perform PUT request
curl_easy_setopt(curl_->handle, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_perform(curl_->handle);

I expected the following

Execute a PUT request to the given server url with an upload range.

Actual behavior

No HTTP request is executed (checked with Wireshark) and instead, the execution of the program is stuck at curl_easy_perform(curl_->handle). The request is only executed if I comment / remove curl_easy_setopt(curl_->handle, CURLOPT_RESUME_FROM_LARGE, resume_from which results in disabling the resume from option.
Note: The request is also executed if I remove the body.

What I have tried to fix the problem

curl/libcurl version

curl-7.83.1
Same problem with curl-7.81.0 which I have used before i encountered the problem.

operating system

Ubuntu 20.04.4 LTS
Linux Simon-PC 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

@bagder bagder added the HTTP label Jun 6, 2022
@bagder
Copy link
Member

@bagder bagder commented Jun 6, 2022

HTTP does not support range "uploads" with PUT or POST.

You can do an implementation, but then you need a server side to agree with the specific method you decide to use from the client and it will not be a generic solution with any HTTP client/server combination.

@simon-berger
Copy link
Author

@simon-berger simon-berger commented Jun 6, 2022

Okay, I understand that.
However, shouldn't curl at least be able to send the request to the server?

Otherwise, that should probably be adjusted in the docs:

For HTTP PUT uploads this option should not be used, since it may conflict with other options. If you need to upload arbitrary parts of a file (like for Amazon's web services) support is limited. We suggest set resume position using CURLOPT_RESUME_FROM, set end (resume+size) position using CURLOPT_INFILESIZE and seek to the resume position before initiating the transfer for each part. For more information refer to https://curl.se/mail/lib-2019-05/0012.html

@bagder
Copy link
Member

@bagder bagder commented Jun 6, 2022

What's the request you want it to send, exactly? Since you're asking for a feature HTTP does not provide it isn't clear to me.

@simon-berger
Copy link
Author

@simon-berger simon-berger commented Jun 6, 2022

I want to send a request like it is described here.

For example say your input file size is 6 and your part size is 4 then
there would be two transfers.
Content-Range: bytes 0-3/*
Content-Length: 4
Content-Range: bytes 4-5/*
Content-Length: 2

@bagder
Copy link
Member

@bagder bagder commented Jun 6, 2022

To upload 4 bytes with a custom Content-Range: header:

$ ./src/curl -d data -H "Content-Range: bytes 0-3/*" -v localhost

It creates this request:

POST / HTTP/1.1
Host: localhost
User-Agent: curl/7.84.0-DEV
Accept: */*
Content-Range: bytes 0-3/*
Content-Length: 4
Content-Type: application/x-www-form-urlencoded

...

@simon-berger
Copy link
Author

@simon-berger simon-berger commented Jun 6, 2022

Ok, many thanks for the answer 👍
So I wrongly assumed that one can use the CURLOPT_RESUME_FROM option for PUT uploads.
Then I think it would be better to remove the section from the CURLOPT_RANGE documentation which describes this to avoid misunderstandings in the future.

@jay
Copy link
Member

@jay jay commented Jun 7, 2022

You have to use CURLOPT_UPLOAD as discussed in the previous e-mail. CURLOPT_CUSTOMREQUEST, "PUT" will not do what you expect:

When you change the request method by setting CURLOPT_CUSTOMREQUEST to something, you do not actually change how libcurl behaves or acts in regards to the particular request method, it will only change the actual string sent in the request.

Regardless maybe I was wrong in the e-mail. You'd have to have a custom read function to stop reading, I think, rather than rely on the default read function in libcurl which will keep reading past the part you want to send.

jay added a commit to jay/curl that referenced this issue Jun 7, 2022
The e-mail link in the advice contains instructions that are prone to
error. We need an example that works and can demonstrate how to properly
perform a ranged upload, and then we can refer to that example instead.

Bug: curl#8969
Reported-by: Simon Berger

Closes #xxxx
jay added a commit that referenced this issue Jun 8, 2022
The e-mail link in the advice contains instructions that are prone to
error. We need an example that works and can demonstrate how to properly
perform a ranged upload, and then we can refer to that example instead.

Bug: #8969
Reported-by: Simon Berger

Closes #8970
@bagder
Copy link
Member

@bagder bagder commented Jun 8, 2022

With @jay's updated man page landed, I think we can close this issue.

@bagder bagder closed this as completed Jun 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants