Can't get simple putStream() script to work...

[TooTallNate]

var s3 = require('s3') // s3 is an initialized knox S3 Client instance
var fs = require('fs')
var file = '/Users/nrajlich/Pictures/avatar.jpg'; // This is a valid JPG file that does exist

var headers = {};
headers['Content-Type'] = 'image/jpeg';

var res = fs.createReadStream(file);
var s3req = s3.putStream(res, '/nate-avatar.jpg', headers, function(err, s3res){
  if (err) throw err;
  console.log(s3res.statusCode);
  console.log(s3res.headers);
  s3res.pipe(process.stdout, {end: false});
  console.error('s3 callback', s3req.url);
});
s3req.on('progress', console.log);

Using knox v0.3.0, this script makes S3 return a 501 error code, with the following XML body:

<Error>
  <Code>NotImplemented</Code>
  <Message>A header you provided implies functionality that is not implemented</Message>
  <Header>Transfer-Encoding</Header><RequestId>7A05E23CB3DBBD1E</RequestId>
  <HostId>Dylx7eIcmXceZCEOZKmMxpGclu2i5bsdlV6pdyH8uAzNm+DAXPE/vdrFkFwFaHht</HostId>
</Error>

Not sure if this is misuse on my part or a problem with knox, but it's kinda holding me up at the moment :)

[domenic]

You need to send Content-Length or use putFile to get it automatically. Without Content-Length Node automatically sets some weird header (Transfer-Encoding: chunked, maybe?) which Amazon rejects.

[rauchg]

@domenic let's put a warning in there if Content-Length is not set

[rauchg]

Unfortunately we need to take it upon ourselves to improve the error reporting at the client layer, because Amazon does a terrible job with that.

[TooTallNate]

We can explicitly disable chunked encoding in node by forcing Connection: close. Getting the "Content-Length" in my case would require Buffering a HTTP request in memory, which I want to avoid.

[rauchg]

So in the absence of Content-Length in the request we should do that then.

[rauchg]

If S3 doesn't support that though, we probably just need to stat the file @TooTallNate and send it along as Content-Length

[TooTallNate]

Forcing "identity" transfer encoding makes the 501 turn into a 403, but it still doesn't work...

var headers = {};
headers['content-type'] = 'image/jpeg';
headers['transfer-encoding'] = 'identity';
headers['connection'] = 'close';

Gives back XML like:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>SignatureDoesNotMatch</Code>
  <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
  <StringToSignBytes>50 55 54 0a 0a 69 6d 61 67 65 2f 6a 70 65 67 0a 46 72 69 2c 20 32 31 20 53 65 70 20 32 30 31 32 20 32 31 3a 35 31 3a 31 32 20 47 4d 54 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 2f 63 6c 6f 75 64 75 70 2d 66 69 6c 60 73 2f 6e 61 74 65 2d 61 76 61 74 61 72 2e 6a 70 67</StringToSignBytes>
  <RequestId>D66D3C6DBC680FFF</RequestId>
  <HostId>1L58FhSdqqis1t5xH87Z5cBS/Rvwn2IVXq6CLa/Xa7E3hSkEAqbEw+AtnmX0e/xy</HostId>
  <SignatureProvided>cHw1NkjS/vzwuU6haCE+Rzj2mMI=</SignatureProvided>
<StringToSign>PUT

image/jpeg
Fri, 21 Sep 2012 21:51:12 GMT
x-amz-acl:public-read
/cloudup-files/nate-avatar.jpg</StringToSign>
  <AWSAccessKeyId>...</AWSAccessKeyId>
</Error>

[domenic]

Notably stat'ing the file to get its Content-Length is exactly what putFile does.

[domenic]

Oh interesting, I wonder if this uncovers a deficiency in the signing algorithm or if Amazon is just giving inaccurate error messages. Hmm!

[TooTallNate]

It would be killer if knox supported a multipart upload handling API (or did this behind the scenes!) http://stackoverflow.com/questions/8653146/can-i-stream-a-file-upload-to-s3-without-a-content-length-header

[domenic]

@TooTallNate pull request welcome!!

[domenic]

Added early error in 41cc7a6 which will be released shortly as 0.3.1.