Add Buffer and Stream support for S3 #3

Closed
tracker1 opened this Issue Dec 5, 2012 · 10 comments

Projects

None yet

9 participants

@tracker1
tracker1 commented Dec 5, 2012

Would be nice to be able to pass a buffer, or stream in addition to a base-64 encoded string. Since these are the typical interfaces for dealing with binary data in Node.js

@reconbot
reconbot commented Dec 5, 2012

+1

@Rob-ot
Rob-ot commented Dec 5, 2012

+1

@trevorrowe
Member

This is a known omission at this point. We will definitely be adding proper support for working with binary data.

@tracker1

Absolutely appreciated... I know supporting a Buffer should be simple enough, and wrapping a stream interface could be a bit more of a pain.

@deanmao
deanmao commented Dec 27, 2012

Ok cool. so it has nothing to do with my code then... I naively assumed I could download files with their s3 library. But there's some weird utf-conversion issue.

@vanb
vanb commented Dec 27, 2012

+1

@deanmao
deanmao commented Dec 28, 2012

I hacked in buffer support in my fork, only because I really needed it, but the code may not be very DRY:

deanmao@994aaec

However, it works... at least I can now download files from S3 and write to disk without having a weird utf-8 conversion screwing up all the bytes.

I could do a pull request, but maybe the library authors prefer the body-as-buffer to be set via a method parameter option instead?

Actually, shouldn't the body always be a buffer until we know we need to convert it to a string? I feel like that whole method could just be done a lot better...

@BharatMeda

@trevorrowe Thank you. Looking forward for this feature.

@lsegal lsegal pushed a commit that closed this issue Jan 4, 2013
Loren Segal Support Buffer & Stream payloads in REST services
The "Body" payload property of REST-based services (like Amazon S3)
will now return Buffer objects instead of Strings, which allows for
the downloading of binary data:

```js
s3.client.getObject({Bucket:'foo',Key:'bar'}, function(err, data) {
  console.log(data.Body); // data.Body is now a Buffer object
});
```

In addition, you can now assign a readable Stream object to the
Body payload:

```js
// Read the body from a stream
var writeParams = {Bucket:'bucket',Key:'image.jpg',Body:stream};
var stream = fs.createReadStream('/path/to/image.jpg');
s3.client.putObject(writeParams, function() {
  // Write the body Buffer object to a file
  var readParams = {Bucket:'bucket',Key:'image.jpg'};
  s3.client.getObject(readParams, function(err, data) {
    var out = fs.createWriteStream('/path/to/image2.jpg');
    out.write(data.Body);
    out.end();
  })
});
```

Conflicts:
  lib/service_interface/query.js

Fixes #3
e0743f1
@lsegal lsegal closed this in e0743f1 Jan 4, 2013
@BharatMeda

Would you be updating the package manager with the latest changes? Thanks in advance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment