In-memory transfer of binary data to S3

[KeithBlonquist]

Can the Aws::StringStream be used to transfer binary data to S3? See the suggested code at the top of issue #64. I am getting a "BadRequest: An error occurred while parsing the HTTP request" error.

[crusader-mike]

No. You could use std::strstream (unless your data length is zero) -- you'll have to live with "this thing is deprecated" message though. Proper way is to write your own implementation of streambuf. I had to spend a day or so before I got minimal version right -- besides linear reading, this SDK moves stream ptr to the end to pre-calc data size.

Be careful, most of internet "streambuf for beginners" guides have bugs in them. :-)

Btw, I suggest not using std::stringstream for any type of files.

[taddygrrr]

I had the same problem trying to upload photo buffers: started with stringstream, later wrote my own StreamBuffer. With StreamBuffer, I'd get the correct content length but the signature of the content was off. I could write my buffer to a temp file then HTTP PUT it because FStream works fine.

Solution: use boost bufferstream. Download but don't build boost. bufferstream and its dependencies are all in headers.

#include <aws/core/Aws.h>
#include <aws/core/utils/memory/stl/AWSStreamFwd.h>
#include <aws/s3/model/PutObjectResult.h>
#include <aws/s3/model/PutObjectRequest.h>
#include <aws/s3/S3Client.h>

#include <boost/interprocess/streams/bufferstream.hpp>

[...]

Aws::Client::ClientConfiguration configuration = Aws::Client::ClientConfiguration();

configuration.region = region;
Aws::S3::S3Client coviS3Client(configuration);

std::string contentLength = std::to_string(bufferLength);
std::shared_ptr<Aws::IOStream> body =  std::shared_ptr<Aws::IOStream>(new bufferstream((char*)buffer, bufferLength));

PutObjectRequest putObjectRequest;
putObjectRequest.WithBucket(bucket).WithKey(keyname).SetBody(body);
return handleResult(coviS3Client.PutObject(putObjectRequest), keyname);

[Sleepyowl]

I wonder why PutObjectRequest requires Aws::IOStream and not just Aws::IStream...

[taddygrrr]

(I think) I figured out my leak issue: curllib on my amazon linux doesn't quite use nss properly for certificates, so it doesn't deallocate memory until app shutdown. We're going to go upgrade Amazon Linux 2 which has an newer version of nss; requires curllib 7.59 or greater, built on top of it--compile time dependency in curllib.

Bug in nss (in curllib's use of nss) :
https://bugzilla.mozilla.org/show_bug.cgi?id=1202413

Fix in curllib:
curl/curl@1605d93#diff-1b39a31e07525713fad76503c7cbc5d1

Thanks.