Avoiding data copy in S3 PutObject flow?

[amirkope]

My application layer is already allocating a memory buffers to which it copies/creates data. Optimally, I would like to serialize and send this data to S3 without having to allocate (and copy to) another buffer in the process. Is there an easy way to avoid buffer allocation and data copy in PutObject flow?

As far as I understand it, in order to use PutObject on an already-in-memory data, I need to create an IOStream from some streambuf object. SDK does offers creating a PreallocatedStreamBuf (so one unnecessary copy is saved) but only from an Aws::Utils::Array, which always allocates its own buffer.

Ideally for me, SDK would expose another kind of PreallocatedStreamBuf that takes a pointer of a memory buffer with no additional overhead, or another kind of Array which can be constructed with a preallocated buffer. In any case, buffer ownership should remain at application layer (which also needs to release it later).

One solution can be to change my application layer to allocate its buffers only via AWS SDK Arrays but I don't really like this solution because my application already has its own memory management and also I would have to change too many layers and APIs along the way.

I can probably do the tedious work of deriving another streambuf myself but I was wondering if I was missing something or maybe there are future plans to expose such API?

Help would be appreciated. Thanks!

[singku]

I think this is what you need:

aws-sdk-cpp/aws-cpp-sdk-core/include/aws/core/utils/stream/ResponseStream.h

Lines 70 to 78 in 970a729

	class AWS_CORE_API DefaultUnderlyingStream : public Aws::IOStream
	{
	public:
	using Base = Aws::IOStream;
	DefaultUnderlyingStream();
	DefaultUnderlyingStream(Aws::UniquePtr<std::streambuf> buf);
	virtual ~DefaultUnderlyingStream();
	};

[amirkope]

Thanks @singku
Actually, the data allocation/copy that bothers me occurs before that stage, in the Aws::Utils::Array class (which can only be constructed while allocating additional memory). I'll try to demonstrate with the following example from TransferManager::DoSinglePartUpload.

Notice that in this code user's stream is duplicated to another stream. The memory copy itself occurs in line 389 where user data is copied to an Array's underlying buffer.

aws-sdk-cpp/aws-cpp-sdk-transfer/source/transfer/TransferManager.cpp

Lines 386 to 391 in cd59bdd

	auto buffer = m_bufferManager.Acquire();
	auto lengthToWrite = (std::min)(static_cast<uint64_t>(buffer->GetLength()), handle->GetBytesTotalSize());
	streamToPut->read((char*)buffer->GetUnderlyingData(), lengthToWrite);
	auto streamBuf = Aws::New<Aws::Utils::Stream::PreallocatedStreamBuf>(CLASS_TAG, buffer, static_cast<size_t>(lengthToWrite));
	auto preallocatedStreamReader = Aws::MakeShared<Aws::IOStream>(CLASS_TAG, streamBuf);

---

I'll try to clarify further by attaching a code snippet, similar to my current code:
userDataBuff is a u8 buffer allocated by the user (application layer). In order to upload userDataBuff to S3, it seems I need to do something like this:

Aws::Utils::Array<u8> array(userDataBuff, userDataBuffSize); // <== data copy inside!
Aws::Utils::Stream::PreallocatedStreamBuf streamBuf(&array, array.GetLength());
auto preallocatedStreamReader = Aws::MakeShared<Aws::IOStream>("preallocatedStreamReader", &streamBuf);
putObjectRequest.SetBody(preallocatedStreamReader);

I'm looking for an option to avoid the data copy in the Array constructor. Two options crossed my mind:

1. Using a different Array that doesn't use allocation/copy (only shallow copy of the buffer pointer).
2. Using a different PreallocatedStreamBuf which uses an underlying buffer of type u8* instead of an Array.

[crusader-mike]

Is there an easy way to avoid buffer allocation and data copy in PutObject flow?

No, you need to write your own streambuf implementation -- it is not easy, but doable. You can probably avoid copy by using std::strstream<>, but it is deprecated.

[taddygrrr]

This issue has been around for some time, see #64. Several people have said write your own streambuf without pointing to one that works. (Have you implemented one, crusader-mike, or did you just read that somewhere?) Best solution I've found is given in #533.

Found that my solution leaks; have narrowed it down to the code in posted in #533, suspect version of SDK I'm using leaks.

[crusader-mike]

Have you implemented one, crusader-mike, or did you just read that somewhere?

from #533:

I had to spend a day or so before I got minimal version right

;)

[taddygrrr]

Have yet to see one that works with the AWS SDK for PUT of a large file to S3.
So, let's see it.

[crusader-mike]

Don't want to deal with HR about making company property public. It isn't hard -- just read related docs carefully.

[taddygrrr]

You could point to any existing StreamBuf implementation online -- there's several that claim to be complete and correct; should people have to write their own streambuf implementation? -- and show how it's supposed to work with AWS SDK. Maybe I'm doing something wrong.

Boost version I posted works. Nothing proprietary about showing code someone else has written blended with AWS SDK examples.

[JonathanHenson]

We could just add an extra constructor to preallocated stream and array if you would like.

[crusader-mike]

You could point to any existing StreamBuf implementation online -- there's several that claim to be complete and correct

We both know how to use google.... ;-) I had to do some bugfixing in one of these implementations lifted off of internet.

Maybe I'm doing something wrong.

I didn't see any leaks in my program (and we moved terrabytes of data already). So, yeah -- chances are you are doing smth wrong. "Whether it is complex enough to be considered an IP" is another sort of conversation I'd like to avoid having with HR -- those guys aren't technical and it won't go well for me.

(Side note: In case if you think I am paranoid -- just read your contract carefully. Most people would never sign a typical IT job contract if they weren't under pressure of "having to make a living". At least I wouldn't.)

[taddygrrr]

It's also worth noting that AWS hasn't posted a complete example showing how to do a PUT of a large buffer to S3. Instead, they show how to upload a file. That works well.

Read the comments on the referenced tickets. @JonathanHenson has suggested a simpler interface on a ticket that's been open since January 3rd, 2016, #64. I'm asking for some follow up.

[JonathanHenson]

The iostream interface can’t be undone, and I take complete responsibility for that design flaw. But in the very near future, we’ll be providing an interface that will make sure you don’t have to ever touch one again if you don’t want to. Basically we have a stream_buf implementation that will run your simpler interface implementation for you automatically.

…

Sent from my iPhone

On Mar 24, 2018, at 1:31 PM, Tad Yeager ***@***.***> wrote: It's also worth noting that AWS hasn't posted a complete example showing how to do a PUT of a large buffer to S3. Instead, they show how to upload a file. That works well. Read the comments on the referenced tickets. @JonathanHenson has suggested a simpler interface on a ticket that's been open since January 3rd, 2016, #64. I'm asking for some follow up. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[crusader-mike]

I hear you, @JonathanHenson -- I burned on iostreams long long (like 15 years?) ago, can't stand them since then (funny to hear this from a diehard C++ fan, isn't it?). That library is a walking can of fat worms. It is better to avoid it at all costs. :-)