Downloading to File or IO object for Aws::S3::Encryption::Client#get_object

[mlangenberg]

The following code example works for Aws::S3::Client, but not for Aws::S3::Encryption::Client.

File.open('filename', 'wb') do |file|
  s3_client.get_object(bucket:'bucket', key:'key', response_target: file)
end

The result is an empty file.

However passing a block seems to be a workaround.

File.open('filename', 'wb') do |file|
  block = -> chunk { file.write(chunk) }
  s3_client.get_object(bucket:'bucket', key:'key', response_target: block)
end

Is this:

• Intended behavior?
• Proper way to stream a file that is encrypted client-side?

[trevorrowe]

This does not look to be intended behavior. It should support an IO object, such as a File. I'll try to look at this shortly.

[trevorrowe]

I've been unable to reproduce this issue. I can read the response into memory, or to a file using the response target path or IO object. Here is what I did:

require 'openssl'
key = OpenSSL::PKey::RSA.new(1024)
s3 = Aws::S3::Encryption::Client.new(encryption_key: key)
s3.put_object(bucket:'aws-sdk', key:'secret', body:'handshake')

resp = s3.get_object(bucket:'aws-sdk', key:'secret')
resp.body.read
#=> "handshake"

s3.get_object(bucket:'aws-sdk', key:'secret', response_target:'./out')
File.read('./out')
#=> "handshake"

File.open('./out2', 'wb') do |file|
  s3.get_object(bucket:'aws-sdk', key:'secret', response_target:file)
end
File.read('./out2')
#=> "handshake"

Can you share an example of what is failing for the encryption client?

[mlangenberg]

You are correct. I cannot reproduce the issue with your code example.

Maybe I was confusing myself with documentation and tutorials using different parameters.

I will close this issue and only reopen it after I run into this again.

[mlangenberg]

One difference between Aws::S3::Client and Aws::S3::Encryption::Client is that get_object does not support passing a block for Aws::S3::Encryption::Client

require 'openssl'
key = OpenSSL::PKey::RSA.new(1024)
s3 = Aws::S3::Encryption::Client.new(encryption_key: key)
s3.put_object(bucket:'aws-sdk', key:'secret', body:'handshake')

s3.client.get_object(bucket:'aws-sdk', key:'secret') do |chunk|
  p chunk #=> "ciphertext"
end

s3.get_object(bucket:'aws-sdk', key:'secret') do |chunk|
  p chunk # never called
end 

[trevorrowe]

I added the block option to #get_object. Please note, using this form will disable retries of failed download. It is not possible for the SDK to know how these byes are consumed and it can not truncate this data sink like it can a file when it has to retry.

Potentially in the future we could explore adding retries with a ranged get, but this is a much greater undertaking.