cannot pass in ::IO objects

[skippy]

https://github.com/aws/aws-sdk-ruby/blob/master/aws-sdk-core/lib/aws-sdk-core/param_validator.rb#L134

the #size method doesn't exist within ::IO (http://ruby-doc.org/core-2.2.3/IO.html). It does within File (which extends IO) and StringIO (which does not) but it isn't within the ::IO object. (which makes sense)

Is there any particular reason why the existence of #size matters?

[awood45]

Size matters because many operations require that the size is known in advance. What are you trying to do?

[awood45]

Size is required from IO-like objects used in our parameters. I don't know of any counter-examples to this. If you're trying to do something where using plain IO without size makes sense, reach out so we can consider the feature request.

[skippy]

hi @awood45

I hear ya. I was attempting to use IO pipes to push data from s3 through to the client without writing a tmp file or putting it in memory. I've used that pattern elsewhere with good effect, but it isn't doable with this library.

The files I'm moving around are in the 10s MB range; so not massive but more than I want to store in memory with a StringIO object. The app is on a system that doesn't have rw access to disk, but it was easy to modify to give the app a scratch space for this purpose.

Anyway, not sure it is worth the pain to do this, but that is/was my use-case.

[trevorrowe]

@skippy I'm curious about your use case. Your original issue made it sound like you wanted to provide an IO object to the #put_object operation and it was rejecting this because it required a size. Your last comment makes it sound like you are trying to download a ~10 mb object from S3 and you down want to buffer it into memory on disk, but rather stream it out.

Amazon S3 requires all calls to PUT object have the total content length specified as a header. This prevents the SDK from being able to stream without buffering the object to compute the total size. It is possible that the SDK could buffer at most 5MB into memory and then switch between a managed multiple upload of 5MB parts or a single PUT object request. However, this would be a new utility that would need to be written.

However, if you are attempting to stream an object out of S3 you can simply call Aws::S3::Client#get_object or Aws::S3::Object#get passing in a :response_target option. This can be an IO object where bytes will be written to. If the response target IO object does not response to truncate then that will disable retries of failed GET requests.

[franzliedke]

@trevorrowe A bit late to the party, but since I just experienced the same issue, I can tell you about my use-case.

We have an in-memory object from rubyzip/rubyzip representing a ZIP file (could be on disk as well). From that, I pull one Zip::InputStream object which references one file in the ZIP archive. Because that class does not implement #size, I cannot upload it directly to S3, having to first call #read on it. Is that a valid use-case? 😀