Downloading Binary Data

[AlexGoico]

Does cpr support downloading binary data to a file? For example how would I download the cookie image in the readme: https://assets-cdn.github.com/images/icons/emoji/unicode/1f36a.png using cpr?

[eigenwhat]

Strictly speaking, no, but the response body is still fully intact in the text attribute of the Response object. You can then write it to a file using the method of your choice.

That said, this kinda sucks because it means the whole file is residing in memory, which is a problem if you try to download huge files. I hope that being able to configure the session to write to a file is something that is planned for a future iteration.

If you want to do some hacking on your own, the cURL options you want to set are CURLOPT_WRITEFUNCTION and CURLOPT_WRITEDATA, and cpr sets them in Session::Impl::makeRequest().

[whoshuu]

@axalon900 is right on.

It's not beyond the capacity of the library to enable some sort direct write-to-file functionality to avoid having to hold data in memory. It can be something as simple as a wrapper around std::ofstream that you put into a request.

[Coderik]

Is there a reason why the feature/download branch has never been merged?

[hwynn]

Is this any closer to being implemented?

I've written the text to files, using std::ofstream, but the result is always messed up (extra line breaks in txt files completely mangled image files).
I believe the python equivalent to what I need is response.iter_content(CHUNK_SIZE) to write binary data to files.

[sepcon]

Is this any closer to being implemented?

I've written the text to files, using std::ofstream, but the result is always messed up (extra line breaks in txt files completely mangled image files).
I believe the python equivalent to what I need is response.iter_content(CHUNK_SIZE) to write binary data to files.

You should open the stream with std::ios::binary and use ostream::write method instead of operator<<

I already implemented the stream support for this but haven't complete the upload functionality and the test cases yet. I will submit the pull request in next few days. if you want to get premature version, I can share you the code first. :)

[hwynn]

Is this any closer to being implemented?
I've written the text to files, using std::ofstream, but the result is always messed up (extra line breaks in txt files completely mangled image files).
I believe the python equivalent to what I need is response.iter_content(CHUNK_SIZE) to write binary data to files.

You should open the stream with std::ios::binary and use ostream::write method instead of operator<<

I already implemented the stream support for this but haven't complete the upload functionality and the test cases yet. I will submit the pull request in next few days. if you want to get premature version, I can share you the code first. :)

Yes I would absolutely like the premature version of the code. I'm scrambling around to find another way to get this functionality in c++. So this should save me a lot of time if I can keep using cpr.

And I would need to use the std::ios::binary as the output stream, correct?

[sepcon]

Is this any closer to being implemented?
I've written the text to files, using std::ofstream, but the result is always messed up (extra line breaks in txt files completely mangled image files).
I believe the python equivalent to what I need is response.iter_content(CHUNK_SIZE) to write binary data to files.

You should open the stream with std::ios::binary and use ostream::write method instead of operator<<
I already implemented the stream support for this but haven't complete the upload functionality and the test cases yet. I will submit the pull request in next few days. if you want to get premature version, I can share you the code first. :)

Yes I would absolutely like the premature version of the code. I'm scrambling around to find another way to get this functionality in c++. So this should save me a lot of time if I can keep using cpr.

And I would need to use the std::ios::binary as the output stream, correct?

Yes, std::ios::binary
you can also checkout and look at stream_tests from my fork branch here:
cpr-feature/stream

[hwynn]

Yes, std::ios::binary
you can also checkout and look at stream_tests from my fork branch here:
cpr-feature/stream
I'm sorry, I don't know what to do with this link. I've never forked a branch before. And this link seems to take me to what looks like a page for creating my own pull request. And the cpr fork on your profile doesn't look like it has any updated files based on the last commit dates.
What am I supposed to do?

[sepcon]

Yes, std::ios::binary
you can also checkout and look at stream_tests from my fork branch here:
cpr-feature/stream
I'm sorry, I don't know what to do with this link. I've never forked a branch before. And this link seems to take me to what looks like a page for creating my own pull request. And the cpr fork on your profile doesn't look like it has any updated files based on the last commit dates.
What am I supposed to do?

Oh, sory,
Please use this link instead https://github.com/sepcon/cpr/tree/feature/Stream
NOTE: by default, you should open std::ofstream or any kinds of std::ostream with std::ios::binary mode. But with text content like a webpage, you don't need to specify that.

[hwynn]

Yes, std::ios::binary
you can also checkout and look at stream_tests from my fork branch here:
cpr-feature/stream
I'm sorry, I don't know what to do with this link. I've never forked a branch before. And this link seems to take me to what looks like a page for creating my own pull request. And the cpr fork on your profile doesn't look like it has any updated files based on the last commit dates.
What am I supposed to do?

Oh, sory,
Please use this link instead https://github.com/sepcon/cpr/tree/feature/Stream
NOTE: by default, you should open std::ofstream or any kinds of std::ostream with std::ios::binary mode. But with text content like a webpage, you don't need to specify that.

Thank you very much. I'm going to try to compile this into a static library (I've been using the static library of cpr from vcpkg until now). Then I'll be able to make it work in my project.

[ingun37]

To anyone who is reading this to find a way to download binary data, there's an undocumented API called Download

https://github.com/whoshuu/cpr/blob/f4622efcb59d84071ae11404ae61bd821c1c344b/include/cpr/api.h#L206-L222

that can retrieve binary data by callback or ofstream.

Here's an example

  auto ofstream = std::ofstream("tmp");
  auto session = cpr::Session();
  session.SetUrl(cpr::Url{url});
  auto response = session.Download(ofstream);

[COM8]

Thanks @ingun37 for taking the time to answer this!

[SpriteOvO]

Is it possible to provide a BinaryWriteCallback for the Download function in the future?

Implementation in my opinion:

class BinaryWriteCallback {
  public:
    BinaryWriteCallback() = default;
    BinaryWriteCallback(std::function<bool(std::vector<unsigned char> data)> callback) : callback(std::move(callback)) {}

    std::function<bool(std::vector<unsigned char> data)> callback;
};

Using std::string for binary data is not very appropriate. Although it works in most cases, and we can convert it to std::vector<unsigned char> ourselves in WriteCallback, it would be better if the library provided it directly. :)