S3 GetObject returns empty result body

[RLThomaz]

The C++ SDK is returning an empty result body for an object that the outcome result IsSucess and has GetContentLenght different than zero.

I followed most of AWS documentation and example codes - including the one officially available at Developer Guide. I also looked around for reported bugs and found this for the AWS JS SDK. Basically, there was something listening and intercepting the 'data' before the SDK could store it in the buffer.

I'm on Ubuntu 16.04, tried everything imaginable (firewall, netstat, killing applications), but still got an empty result body. I was not able to fix it - thus I'm submitting this issue - but I found a workaround. My workaround for this 'bug' uses the SetResponseStreamFactory to write a file as soon as the object is downloaded from the bucket.

objectRequest.SetResponseStreamFactory([filePath]()
{
 return new std::fstream(filePath.c_str(), std::ios::out | std::ios::binary);
});

[singku]

Hello, is this something always happens? Could you please give more details about how it is empty? A sample code helps.

Also could you please try to build and run following code example?

src/CMakeLists.txt

cmake_minimum_required(VERSION 3.2)
project(SDKTest)
SET(CMAKE_CXX_STANDARD 11)

# On Windows, AWSSDK_DIR should be provided to Cmake
if (WIN32)
        SET(AWSSDK_DIR "C:/Program Files/aws-cpp-sdk-all/lib/cmake/AWSSDK")
endif()

find_package(AWSSDK REQUIRED)

Message(STATUS "AWSSDK_LIB_DIR ${AWSSDK_LIB_DIR}")
Message(STATUS "AWSSDK_BIN_DIR ${AWSSDK_BIN_DIR}")
Message(STATUS "AWSSDK_INCLUDE_DIR ${AWSSDK_INCLUDE_DIR}")

SET(SERVICE "s3")
AWSSDK_DETERMINE_LIBS_TO_LINK(SERVICE OUTPUT)
Message(STATUS "All linked libs: ${OUTPUT}")

link_directories("${AWSSDK_LIB_DIR}")
add_executable(SDKTest ../src/s3.cpp)

if (WIN32)
    AWSSDK_CPY_DYN_LIBS(SERVICE "" ./)
endif()

target_link_libraries(SDKTest ${OUTPUT})
target_include_directories(SDKTest PRIVATE ${AWSSDK_INCLUDE_DIR})

src/s3.cpp

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/GetObjectRequest.h>
#include <aws/s3/model/PutObjectRequest.h>

using namespace Aws::S3;
using namespace Aws::S3::Model;

static const char* const KEY = "SOME_KEY_NAME";
static const char* const BUCKET = "SOME_BUCKET_NAME";

int main()
{
    Aws::SDKOptions options;
    options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;

    Aws::InitAPI(options);
    {
        Aws::Client::ClientConfiguration config;
        S3Client s3Client(config);

        auto requestStream = Aws::MakeShared<Aws::StringStream>("S3Upload");
        *requestStream << "Hello from the S3 Client!";

        PutObjectRequest putReq;
        putReq.WithBucket(BUCKET).WithKey(KEY).SetBody(requestStream);

        auto putOutcome = s3Client.PutObject(putReq);
        if (putOutcome.IsSuccess())
        {
            std::cout << "PutObject success!" << std::endl;
        }
        else
        {
            std::cout << "PutObject failed!" << std::endl;
            exit(-1);
        }

        GetObjectRequest req;
        req.WithBucket(BUCKET).WithKey(KEY);

        auto outcome = s3Client.GetObject(req);
        if (outcome.IsSuccess())
        {
            std::cout << "Get object success! " << outcome.GetResult().GetContentLength() << std::endl;
            std::cout << outcome.GetResult().GetBody().rdbuf() << std::endl;
        }
        else
        {
            std::cout << "Get object failed! " << outcome.GetError().GetMessage() << std::endl;
        }
    }
    Aws::ShutdownAPI(options);

    return 0;
}

[RLThomaz]

Hello, is this something always happens?

Hello, I just started using the AWS SDK, so always would be a bit too strong here. But it happened to all my files on my buckets.

Your code example works - even using my bucket and files. So I assumed there was something missing on CMake, but I used my own version and it still works just fine. So I'm clearly missing something here, but it is still weird that my workaround works.

Could you please give more details about how it is empty?

It was really empty - using a breakpoint and taking a look inside GetResult() and GetBody() would show that the object had a length but nothing inside the buffer. I also monitored the network and the data was being downloaded.

A sample code helps.

I tried to get the response body exactly the same way you did on your example (notice that myFile is a Aws::OFStream):

if (outcome.IsSuccess())
{
  std::cout << "Get object success! " << outcome.GetResult().GetContentLength() << std::endl;
  myFile << outcome.GetResult().GetBody().rdbuf() << std::endl;
}

So, the major difference from your code to mine is that I download the object inside a static method of a class. Therefore, my main.cpp looks something like:

int main(int argc, char const *argv[])
{
  Aws::SDKOptions defaultOptions;
  AWS::InitAPI(defaultOptions);
  {
    MyNamespace::MyClass::myStaticDownload(bucket,keyname);
  } Aws::ShutdownAPI(defaultOptions);
}

And my class looks something like:

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/GetObjectRequest.h>

namespace MyNamespace
{
  class MyClass
  {
    static void myStaticDownload(Aws::String bucket, Aws::String keyname)
    {
      ...
      if (outcome.IsSuccess())
      {
        std::cout << "Get object success! " << outcome.GetResult().GetContentLength() << std::endl;
        myFile << outcome.GetResult().GetBody().rdbuf() << std::endl;
      }
    }
  }
}

I will try to go back to "GetBody().rdbuf()" and see if it was suddenly fixed for my computer.

EDIT -------
Tested again using GetResult().GetBody().rdbuf() and it is still empty.

.
.
.
Aws::OFStream myFile;
myFile.open(filePath.c_str(), std::ios::out | std::ios::binary);
myFile << outcome.GetResult().GetBody().rdbuf();

std::cout << filePath.c_str() << " returned " << outcome.GetResult().GetContentLength() << " bytes"
               << "\t and the body has " << outcome.GetResult().GetBody().gcount() << " bytes." << std::endl;
.
.
.

So this returns:

MyFile.ply retruned 578632 bytes and the body has 0 bytes.

[crusader-mike]

Wouldn't first line in your code below "drain" the buffer making it empty?

myFile << outcome.GetResult().GetBody().rdbuf();
...
std::cout << " returned " << ... outcome.GetResult().GetBody().gcount() ...

Now here you simply aren't honest with us:

So this returns:
MyFile.ply retruned 578632 bytes and the body has 0 bytes.

.. your code can't print "retruned"

[RLThomaz]

Wouldn't first line in your code below "drain" the buffer making it empty?

No, because it is already empty EDIT: I know this because I debugged and checked the object - in addition to the file being empty. However, I read about what outcome.GetResult().GetBody().gcount() does and it only counts what was actually read from the buffer. So it should not mean that "the body has 0 bytes", but that "0 bytes were read from the stream".

Now here you simply aren't honest with us:
.. your code can't print "retruned"

I was being honest, I did not copy and paste the return because the terminal closed as soon as the code finished, so I typed it manually here for the sake of the example. This typo does not change, however, what was actually printed - and my remark about what gcount() actually does.

I will come back to this later.

[crusader-mike]

No probs, I was just messing with ya ;-)

Something isn't right here... I don't know every details about iostreams as I hate them and walk extra mile to avoid them. But gcount() doesn't necessarily seem to return stream "size" if you check it's description.

Why redirecting streambuf (rdbuf() result) to std::ofstream produces empty file -- beats me, but I bet there is a simple explanation for it. Just debug it, it shouldn't be hard to figure it out.

[RLThomaz]

But gcount() doesn't necessarily seem to return stream "size" if you check it's description.

Yes, that was what I meant by

So it should not mean that "the body has 0 bytes", but that "0 bytes were read from the stream".

Still, it is showing us that the last operation (rdbuf()) read 0 bytes - because there were 0 bytes left to be read (I suppose).

I will debug everything again and remove those from the static method (don't know if this makes any sense, however).

[crusader-mike]

Still, it is showing us that the last operation (rdbuf()) read 0 bytes

but rdbuf() doesn't read any bytes -- it just returns a pointer to internal buffer. All reading happens in related operator<<, probably in the loop. All you could say that last iteration in that loop read 0 bytes, which is probably normal.

[crusader-mike]

also, maybe reading directly from streambuf doesn't update stream's gcount.

[RLThomaz]

Thanks for the help @singku and @crusader-mike - I have found this interesting, yet simple, stack overflow question. It seems that the stream needs the std::ios::in flag in order to get the data from the buffer.

I tested it out with another (unrelated) code and indeed an ostream.rdbuf() was always empty until I created it using the std::ios::in flag (in addition to the out flag).

I will investigate if the outcome.GetResult().GetBody() stream is open with in and out.

EDIT --------
Found an interesting comment on one of the test files.

getObjectRequest2.SetResponseStreamFactory(
            [=](){ 
                // NOTE: If using an FStream in order to download a file from S3 to a physical file, then we need to specify
                // the filemode "std::ios_base::out | std::ios_base::in | std::ios_base::trunc" --
                // If the file transfer fails, then the error stream from the httpRequest is written to the file instead of the
                    // actual file contents.
                // If this is the case, then the BuildAWSError function assumes it can 'read' the stream in order to read in the
                // error status and create an AWSError.  If the file is not marked as read/write (or write only), this will fail
                // and the error status will instead be an XML_PARSE_ERROR error because the BuildAWSError function failed
                // The 'truncate' is required to ensure that if the file download IS successful, then it can be written to the
                // FStream (if ::trunc is not specified, then the FStream.write fails for some unknown reason)

                return Aws::New<Aws::FStream>(ALLOCATION_TAG, TestFileName.c_str(), std::ios_base::out | std::ios_base::in | std::ios_base::trunc); 
            }
        );

[crusader-mike]

yes, I recall these lines: they were the reason why I ended up using multiple GetObject(range) calls instead of single GetObject(custom stream) -- I've realized that I'll have to implement custom streambuf that supports both input and output (instead of just output). So, instead of having to deal with iostream's crazy logic I ended up with a bit sub-efficient code. :)