Support partial reads in storage::Client::ReadObject.

[coryan]

We need to support partial reads in storage::Client::ReadObject (i.e. reading a range of bytes from the object). This will allow application developers to download the file "in parallel" with multiple threads reading different sections of the object.

In principle something like:

void F(google::cloud::storage::Client client) {
  // Read the first 32MiB 
  auto stream = client.ReadObject(
      "my-bucket", "my-object", google::cloud::storage::Range(0, 32 * 1024, 1024));
  while(not stream.eof()) {
    // Do stuff...
  }
}

[benjaminr]

Happy to have a look at this one.

[coryan]

@benjaminr that sounds great, just sends us a PR. As you might expect, you will need to sign a CLA. We have instructions on how to setup your development environment and how to locally execute builds with the same configuration as the CI builds.

Please do not hesitate to @-mention me in this bug if you need help.

[benjaminr]

@coryan Shall do! Thanks for the links and information.

[benjaminr]

@coryan Would it be sufficient to overload ReadObject to something like the following:

c++  
template <typename... Options>
  ObjectReadStream ReadObject(std::string const& bucket_name,
                              std::string const& object_name,
                              std::int64_t begin,
                              std::int64_t end,
                              Options&&... options) {
    internal::ReadObjectRangeRequest request(bucket_name, object_name, begin, end);
    request.set_multiple_options(std::forward<Options>(options)...);
    return ObjectReadStream(raw_client_->ReadObject(request).value());
  }

Where begin and end are passed to internal::ReadObjectRangeRequest?

[coryan]

Unfortunately that overload will not work, because it is ambiguous to use it like this:

void F(gcs::Client client) {
  auto reader = client.ReadObject("my-bucket", "my-object", 0, 2048);
}

The conversion from int to std::int64_t is worse than binding to the parameter pack in the first overload, and the compiler gets all confused. Of course the user can say:

void F(gcs::Client client) {
  auto reader = client.ReadObject("my-bucket", "my-object", std::int64_t(0), std::int64_t(2048));
}

and that works fine. But it seems too surprising that the overload works with some integral types but not with others.

That is why I thought we would need a new Range option:

struct google::cloud::storage:Range : public internal::ComplexOption<Range, RangeData> {
  Range(std::int64_t begin, std::int64_t end);
  // look at DisableMD5Hash for inspiration...
};

Which would be applied to the ReadObjectRangeRequest:

class ReadObjectRangeRequest
    : public GenericObjectRequest<
          ReadObjectRangeRequest, DisableCrc32cChecksum, DisableMD5Hash,
          EncryptionKey, Generation, IfGenerationMatch, IfGenerationNotMatch,
          IfMetagenerationMatch, IfMetagenerationNotMatch, Range, UserProject> {
// delete the begin_ and end_ fields, use `HasOption<Range>()` and `GetOption<Range>()` instead

I think that would work better, but I probably have some details wrong. Does it make sense?

[benjaminr]

Yeah I think that makes sense. In the following struct declaration:

struct google::cloud::storage:Range : public internal::ComplexOption<Range, RangeData> {
  Range(std::int64_t begin, std::int64_t end);
  // look at DisableMD5Hash for inspiration...
};

Is RangeData best as a std::pair for begin and end? Apologies if this is obvious, my C++ skills are extremely rusty. Thanks!

[coryan]

Both work for me, a pair looks easier.

[coryan]

@benjaminr I assigned this to you just so we can keep track of who is working on what, if at any point you think you won't be able to work on it anymore just let me know.

[benjaminr]

@coryan you might be better assigning to someone else this time as I'm not sure when I'm going to get the time to work on it at the moment. Apologies for the delayed response.

[coryan]

@benjaminr no problem, I hope you had fun looking at this issue at least. After thinking more about it, it is not that easy because we need to disable checksums when reading a section of the file.