Feature request: Retrieving data from response headers when streaming S3 objects

[janko]

In my open source project I'm using Aws::S3::Object#get with a block to stream the S3 object on-demand:

chunks = object.enum_for(:get, **options)
chunks #=> #<Enumerator>
chunks.next # first chunk of data
chunks.next # next chunk of data

I'm wrapping this functionality into a custom IO-like object, so that people can stream using the IO#read semantics, which under the hood retrieves subsequent chunks of data when needed.

I'd like the content length to be known at the start of streaming, which would be returned when the user called #size. Currently I have to make an additional #head_object request to retrieve Aws::S3::Object#content_length. However, the #get_object request that Object#get makes already retrieves response headers before it starts streaming, which I think hold all the same information that response headers for a #head_object call would hold.

So, it would be convenient if Object#content_length, #content_type and other data was filled in on Object#get as soon as the response headers are received (before the response body is read). That way I could avoid the additional HEAD request to S3 for this use case, to retrieve data that we technically already have, but which can only be retrieved once the whole S3 object has been downloaded.

In code, I would like this behaviour:

object.get do |chunk|
  # this currently makes an additional HEAD request, but doesn't have to,
  # because it could fill this info in from the response headers of the
  # `#get_object` call that we already retrieved at this point
  object.content_length
end

I think it should be possible to somehow use the notification about response headers that Seahorse transmits:

aws-sdk-ruby/gems/aws-sdk-core/lib/seahorse/client/net_http/handler.rb

Line 80 in e5ef638

resp.signal_headers(status_code, headers)

[srchase]

Tracking as a feature request: FEATURE_REQUESTS

[mullermp]

Reopening - deprecating usage of Feature Requests backlog markdown file.

[janko]

Thanks. FWIW, I've recently managed to get the desired behaviour with the following hack:

req = client.build_request(:get_object, **params, bucket: bucket.name, key: object.key)

body = req.enum_for(:send_request)
body.peek # start the request

content_length = Integer(req.context.http_response.headers["Content-Length"])
chunks         = Enumerator.new { |y| loop { y << body.next } }

[alextwoods]

I've been looking into how we might generically achieve this for streaming methods. One option I see is to add an additional, optional parameter to the Proc that is being called which provides the headers, eg:

client.get_object(bucket: bucket_name, key: key) do |chunk, headers|
  content_length = Integer(headers['content-length'])
  # do something with chunk
end

You mention that you're wrapping this into a custom IO like object to provide IO#read - another option could be adding this behavior to the SDK (possibly as a third type of response_target) as it sounds like this could be useful for our users.