Provide streamed access to object data

[jboynes]

The BlobReadChannel and BlobWriteChannel implementations buffer data and perform separate RPC calls when they need to flush the buffer. This allows for retries of the transfer operations but uses more billable operations than just writing the data in a continuous stream.

[aozarov]

I am not sure I understand the issue.

1. write-channel only buffer to the minimum block size (256K bytes - see https://cloud.google.com/storage/docs/concepts-techniques#resumable)
2. read is reading as much as requested or 256K (for minimum transfer per request).

The underlying API for the service (apiary or XML) does not support streaming RPC, so every write/fetch has to be done via a separate RPC call. I expect that to change in the future when the service will support gRPC. Is that what the issue about?

[aozarov]

Clarification would be helpful but now I think you were probably referring to the non-resumable write option (e.g. if you were using the apiary client directly you would do something like insertObject.getMediaHttpUploader().setDirectUploadEnabled(true))? And similarly for readers.

If so, then I think we should split the issue to reader and writer.

As for writers it is highly recommended (and also mandated in AE classic) to use resumable writes when content is not small (some examples suggest < 2MB, AE currently limit is 10MB). When the content is small one could write it via the create method (and because the content is small the value of Stream over byte[] is not that meaningful).

As for readers, I think that recommendation is weaker (and AE class constraint is wider 32MB) but still
exists mostly for error handling issues (as you pointed out) and based on my experience when building appengine-gcs-client errors handling and retries made a big difference in the user experience. I guess I could have change "byte[] load"
to return a stream (or to have some way to distinct between a directDownload to resumable/chunk download in the same call though in the case of the former the returned channel/stream would not be serializable) but considering the underlying Apiary client code (https://github.com/google/google-api-java-client/blob/86fd2a574487aaff9b4dbdd0396a98bb2741828f/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpDownloader.java#L181 & https://github.com/google/google-api-java-client/blob/86fd2a574487aaff9b4dbdd0396a98bb2741828f/google-api-client/src/main/java/com/google/api/client/googleapis/media/MediaHttpDownloader.java#L247) I don't think this is going to be useful as it seems that in the case of directDownload bytes are only transferred as once request was completed.

[jboynes]

Something like:

  /**
   * Returns a stream that can be used to read a stored object.
   *
   * @param uri the id of the object to be read
   * @param options options specifying how the object is to be opened; defaults to READ
   * @return a stream for reading the object
   * @throws IOException if a problem occurred accessing the object
   */
  InputStream newInputStream(URI uri, OpenOption... options) throws IOException;

  /**
   * Returns a stream that can be used to write an object.
   *
   * @param uri the id of the object to be written
   * @param options options specifying how the object is to be opened;
   *  defaults to CREATE, TRUNCATE_EXISTING, WRITE
   * @return a stream for writing the object
   * @throws IOException if a problem occurred accessing the object
   */
  OutputStream newOutputStream(URI uri, OpenOption... options) throws IOException;

which provides an idiomatic, stream-based way to upload and download object content. These would map to GET and POST operations that allow the content to be transferred with a single exchange.

[aozarov]

See my reply above.
Considering the current state of the Apairy client I see no reason for having these method (though
we could by pass some of the issues by using the XML API instead).
Also, these methods are hiding many GCS specific options or differences (such as passing Object
metadata in write operation, conditional read/write, lack of append mode,...) and therefore I don't think
their place is in this library, however I do think a similar functionality would be nice/useful as part of the
nio.file implementation in a contrib library (as discussed in #46) .

[jgeewax]

Hmm, I think we should definitely have Input/Output Stream methods. This is how AWS does it, not to mention gcloud-node does this as well.

[aozarov]

Channel is a stream. It is part of the newer java nio API. see my comment in #46.

[jgeewax]

So you're saying that we already have this functionality? What's the gcloud-java equivalent of

AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());        
S3Object object = s3Client.getObject(
                  new GetObjectRequest(bucketName, key));
InputStream objectData = object.getObjectContent();
// Process the objectData stream.
objectData.close();

[aozarov]

I think the main discussion here is about a request for a stream based one RPC call (non-serialize/non-resumeable) vs current state which provides a serializable and resumeable stream based on multiple resumable RPCs. I agree that if that is indeed the topic discussed here the title is wrong as we do provide a "streamed access to object data".

[aozarov]

see DownloadAction in the storage example: https://github.com/GoogleCloudPlatform/gcloud-java/blob/master/src/main/java/com/google/gcloud/examples/StorageExample.java

        try (BlobReadChannel reader = storage.reader(blob.bucket(), blob.name())) {
          WritableByteChannel channel = Channels.newChannel(writeTo); 
          ByteBuffer bytes = ByteBuffer.allocate(64 * 1024);
          while (reader.read(bytes) > 0) {
            bytes.flip();
            channel.write(bytes); // or do whatever with the bytes read.
            bytes.clear();
          }
        }

[aozarov]

And if for any reason one prefers not to use java.nio and use InputStream instead he can do:

InputStream input = Channels.newInputStream(storage.reader(blob.bucket(), blob.name()));

[jboynes]

Channels are not as well supported as streams and there is no downside to providing both.

    Path localData = Paths.get("localData.txt");
    try (OutputStream os = storage.newOutputStream(TEST_OBJECT)) {
      Files.copy(localData, os);
    }

That's setting aside the implementation difference where the Blob channel performs multiple (billed) operations against GCS rather than just one.

Also BlobReadChannel is wrapping HttpResponse objects that can already provide a InputStream over their content directly.

[aozarov]

1. I would replace "supported" with "adopted" and even that I might not fully agree with.
   Given that it is very easy to get an InputStream from a Channel (using a standard API)
   I don't see a reason to provide both. However, I don't mind to provide an InputStream instead of a Channel (that is a minor change with a minor [nagtive in my mind] value, though the negative part
   is slightly higher with OutputStream vs writer Channel as Buffers that go with Channels are more widely accepted).

2. So, it seems that there are various issues discussed here. Please decide what this topic this issue was create for and create other issues for the other (related) topics.

3. As mentioned above (multiple times), the apiary client does not expose the HttpRespose stream directly.

[aozarov]

One more data point to the nioChannel vs io.Stream discussions (which is minor in my mind as they are both "streams" can be trivially converted from one another) is that it looks like grpc-java is based on Netty which is based on nio. I am not sure how much of the nio would be visible from the grpc layer but if something does it is more likely to efficiently work with other nio related classes.

[aozarov]

@jgeewax I hope this addressed your concern. We do provide a streaming API to read a content of a Blob and we are using the newer Java standard for it (I assume Amazon API preceded nio but that
should not be a factor). Converting between the two APIs is trivial and provided by the Java standard APIs. I consider this issue to be nio.Channel vs io.Stream and I plan to close as "wontfix".
The more I think about it the more I am convinced that providing the Channel would be preferred.
Here are some of my reasoning:

1. The nio is the new Java IO API and there is no good reason to stick with the older one (newer framework, libraries are more likely to support the latter - also see (2)).
2. Channels are provided with both Synchronous and Asynchronous flavors for when we provide an asynchronous API calls.
3. App Engine GCS client API which has significant number of users (and traffic) is also using Channels, so migration will be smother.

[jgeewax]

OK if this is really just a difference between nio.Channel and io.Stream it sounds like it's "easy enough" for people to convert between the two. Some more questions...

• Do we have any idea of how annoying this is to external people?
• Are we shoving them in a direction that we need not necessarily shove them?
• Would we incur additional maintenance cost by virtue of having these methods tossed in for convenience?

@jboynes , any opinions on these? @aozarov ?

[aozarov]

I don't think this will "annoy" people. nio was part of Java 7 (which this library specify as minimum requirement) and Java 7 is out there since 2011. That was never a complaint by people that were using the appengine_gcs_client.

I think using the channel is the right direction for the reasons I mentioned above (and the conversion is a trivial one-liner) and for that reason I don't think we should pollute the API with 2 flavors that provide the same, and only channels would be able to support the async option in the future.

I am closing this issue (feel free to re-open if you think it should be further discussed) and I created issue #72 for using one RPC for read-write vs using chunks (multiple resumable RPC calls).