Authenticating for S3 with hardcoded credentials doesn't work in PhantomJS

[lavelle]

I've written an application that connects to a single, user-selected bucket, using that user's key and secret to authenticate. My authentication code is essentially just taken from the introductory guide.

It works fine in my application (a Chrome extension), but when I try to run the same code in a PhantomJS headless Webkit instance with grunt-mocha, my initial call to listObjects fails with the error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. The credentials work in Chrome and are being loaded from the same config file for the tests. I've triple-checked that they're correct, and there's no other differences between my app and test code. The bucket name and region are also identical.

Any ideas about what could cause this?

[lsegal]

Unfortunately, we don't quite fully support PhantomJS. That said, we do run a few integration tests using S3 in PhantomJS, and I have never seen that signature failure before. I would point out that if you had credentials misconfigured you would not get a signing error, but an auth error. The SignatureDoesNotMatch error implies that something is going wrong in the SDK (probably due to lack of PhantomJS support).

Can you provide more details about that error? Knowing the parameters, possibly even the httpRequest itself, would be helpful to reproduce.

Also, what version of PhantomJS are you using?

[lavelle]

I'm using Grunt Mocha version 0.4.11, which uses PhantomJS 1.9.0-1.

The call is

s3.listObjects({ Bucket: 'my-bucket' }, function(error, data) {
    if (error) {
        throw error;
    }
});

Do I need to dig into the library internals to get a log of the HTTP Request?

[lsegal]

The HTTP request is available on the request object, which is available from the this context in the callback:

s3.listObjects(params, function(err, data) {
  console.log(this.request.httpRequest)
});

FWIW I am using PhantomJS 1.9.7. Have you tried using this version for your tests?

[lavelle]

I got the same error with 1.9.7.

Here's the HTTP request.

{
  method: 'GET',
  path: '/',
  headers: {
    'X-Amz-User-Agent': 'aws-sdk-js/2.0.5',
    'Content-Type': 'application/octet-stream; charset=UTF-8',
    'Content-Length': 0,
    Host: 'chromeostest.s3-us-west-2.amazonaws.com',
    'X-Amz-Date': 'Mon, 21 Jul 2014 20:31:44 GMT',
    Authorization: 'AWS AKIAJAAMWRX6TZ72MR2A:QlyErRA0zOPGCPZQEoNUq7eaw3Y='
  },
  body: '',
  endpoint: {
    protocol: 'https:',
    host: 'chromeostest.s3-us-west-2.amazonaws.com',
    port: 443,
    hostname: 'chromeostest.s3-us-west-2.amazonaws.com',
    pathname: '/',
    path: '/',
    href: 'https://s3-us-west-2.amazonaws.com/',
    constructor: [Object]
  },
  region: 'us-west-2',
  virtualHostedBucket: 'chromeostest',
  stream: {
    readyState: 4,
    response: '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>47 45 54 0a 0a 0a 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 4d 6f 6e 2c 20 32 31 20 4a 75 6c 20 32 30 31 34 20 32 30 3a 33 31 3a 34 34 20 47 4d 54 0a 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 3a 61 77 73 2d 73 64 6b 2d 6a 73 2f 32 2e 30 2e 35 0a 2f 63 68 72 6f 6d 65 6f 73 74 65 73 74 2f</StringToSignBytes><RequestId>68C94C74ACB805BD</RequestId><HostId>Q+1NSJaI505LCLdHXYGn9gk2yT/f0MClPEhMfb+cFsC3EKqWeeZntXcxlG7vy5Wf</HostId><SignatureProvided>QlyErRA0zOPGCPZQEoNUq7eaw3Y=</SignatureProvided><StringToSign>GET\n\n\n\nx-amz-date:Mon, 21 Jul 2014 20:31:44 GMT\nx-amz-user-agent:aws-sdk-js/2.0.5\n/chromeostest/</StringToSign><AWSAccessKeyId>AKIAJAAMWRX6TZ72MR2A</AWSAccessKeyId></Error>',
    responseXML: [Object],
    onload: null,
    onerror: null,
    onloadstart: null,
    status: 403,
    onabort: null,
    upload: [Object],
    onreadystatechange: null,
    responseType: '',
    onprogress: null,
    withCredentials: false,
    responseText: '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>47 45 54 0a 0a 0a 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 4d 6f 6e 2c 20 32 31 20 4a 75 6c 20 32 30 31 34 20 32 30 3a 33 31 3a 34 34 20 47 4d 54 0a 78 2d 61 6d 7a 2d 75 73 65 72 2d 61 67 65 6e 74 3a 61 77 73 2d 73 64 6b 2d 6a 73 2f 32 2e 30 2e 35 0a 2f 63 68 72 6f 6d 65 6f 73 74 65 73 74 2f</StringToSignBytes><RequestId>68C94C74ACB805BD</RequestId><HostId>Q+1NSJaI505LCLdHXYGn9gk2yT/f0MClPEhMfb+cFsC3EKqWeeZntXcxlG7vy5Wf</HostId><SignatureProvided>QlyErRA0zOPGCPZQEoNUq7eaw3Y=</SignatureProvided><StringToSign>GET\n\n\n\nx-amz-date:Mon, 21 Jul 2014 20:31:44 GMT\nx-amz-user-agent:aws-sdk-js/2.0.5\n/chromeostest/</StringToSign><AWSAccessKeyId>AKIAJAAMWRX6TZ72MR2A</AWSAccessKeyId></Error>',
    statusText: 'Forbidden'
  }
}

I logged it with depth=1 to avoid the circular structures but if you need the contents of upload, constructor or responseXML let me know.

[lsegal]

It looks like PhantomJS strips Content-Type out of a request even if provided by the client in a GET (and possibly HEAD) request. You can make a request work by doing:

var req = s3.listObjects(params);
req.on('build', function() {
  if (req.httpRequest.method === 'GET') {
    delete req.httpRequest.headers['Content-Type'];
  }
});
req.send(function(err, data) {
  console.log(err, data);
});

I'm not sure what the fix here is, since no other browser I know of strips out Content-Type in GET requests. We could try to strip it out on the customer's behalf, but this might break other browsers that always try to send through a Content-Type (I haven't checked if any browsers do this).

[lavelle]

Fantastic, that fixed it! Thanks a lot for your help.

I personally don't mind about this -- I'm only using PhantomJS for testing, so having some extra code to fix it isn't a problem as it doesn't affect production code at all.

Hope you find a way to fix it anyway though, sure it'd still be useful for some people.

[lsegal]

Looks like this seems to work in most browsers. I'm going to test this through in IE* but I don't expect this to be problematic. Thanks for reporting.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.