s3.listObjects() is Suddenly returning Null

[justinpeake]

My s3.listObjects() is suddenly returning a null data object when I run locally. There is no "permission denied" console message from AWS. Similarly, I am no longer able to access my bucket via Cyberduck. When I enter the appropriate access key id, and secret key... it still says "login failed".

This has only been happening for a week. The app was functioning perfectly since Feb. I have not altered the .env (where access keys are stored).

STRANGELY, the same app code is running perfectly on Heroku, no errors, and the s3.listObjects() is returning an array of filenames. As I mentioned... everything was working very well until last week.

Thanks for your help... I'm kind of stumped.

Here is the error:

node_modules/aws-sdk/lib/request.js:30
throw err;
^

TypeError: Cannot read property 'Contents' of null
at Response. (/Volumes/ARTICULATED/figuresApp passporthack/app.js:329:35)
at Request. (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:354:18)
at Request.callListeners (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
at Request.emit (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:596:14)
at Request.transition (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:21:10)
at AcceptorStateMachine.runTo (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:37:9)
at Request. (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:598:12)
at Request.callListeners (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at Request.emit (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
at Request.emit (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:596:14)
at Request.transition (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/request.js:21:10)
at AcceptorStateMachine.runTo (/Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /Volumes/ARTICULATED/figuresApp passporthack/node_modules/aws-sdk/lib/state_machine.js:26:10

[chrisradek]

@justinpeake
What version of the SDK are you using, and is it the same as what is being used with Heroku?

[justinpeake]

@chrisradek
SDK 2.4.0 and Heroku is using 2.3.17...
I'm realizing now that Heroku has my config variables stored on its own. Which might indicate a problem with my .env file. BUT that wouldn't really explain why the Cyberduck login is failing.

[chrisradek]

I can't really explain why Cyberduck would be failing, but can you try using the config variables that Heroku uses with your local test? If that doesn't work, can you try downgrading your SDK locally to match what's in Heroku?

This seems more like an issue with config. Cyberduck doesn't use the node.js SDK to connect to S3 (it's written in Java), so differences between versions of the SDK shouldn't be relevant.

[justinpeake]

The config variables are identical. Just downgraded versions and tested with same results. That's what's so puzzling is the same config vars working in one place and not working in two other instances.

[chrisradek]

I should have asked before, what does the error returned to your callback say? When data is null, error should be populated.

[justinpeake]

Ah thanks for asking, I hadn't thought to do that, either.

{ [TimeoutError: Missing credentials in config]
message: 'Missing credentials in config',
code: 'CredentialsError',
time: Wed Jun 15 2016 14:17:48 GMT-0400 (EDT),
originalError:
{ message: 'Could not load credentials from any providers',
code: 'CredentialsError',
time: Wed Jun 15 2016 14:17:48 GMT-0400 (EDT),
originalError:
{ message: 'Connection timed out after 1000ms',
code: 'TimeoutError',
time: Wed Jun 15 2016 14:17:48 GMT-0400 (EDT) } } }

[chrisradek]

Thanks. So by default when using node.js, the SDK will try to resolve credentials using the following order:

1. If credentials are configured directly in code, use those otherwise...
2. If they are provided as ENV variables, use those otherwise...
3. If they exist in the shared credentials file, use those otherwise...
4. If you're running in an EC2 instance, call the metadata service to get credentials.

So the heroku app is getting credentials somewhere in this chain. Can I ask where you are currently defining your creds? We can also take a look at what the configured creds look like at the time you made a request to see if there's a discrepancy there as well.

[justinpeake]

I have them as ENV variables... like this:

var AWS_ACCESS_KEY = process.env.AWS_ACCESS_KEY;
var AWS_SECRET_KEY = process.env.AWS_SECRET_KEY;
var S3_BUCKET = process.env.S3_BUCKET;

I can console.log these variables and the strings are there.

I tried putting them directly in code as well (just to test if it was a variable problem) and I was getting the same issue.

[justinpeake]

This is the snippet:

aws.config.update({accessKeyId: AWS_ACCESS_KEY, secretAccessKey: AWS_SECRET_KEY});

              var s3 = new aws.S3();

              var folder = userID + "/";

              console.log(folder);

              var s3_params = {
                  Bucket: S3_BUCKET,  

                  Key: folder, 

                  Expires: 60,
                  ACL: 'public-read'  
              };

              s3.listObjects({Bucket: S3_BUCKET, Delimiter: '/', Prefix: folder}, function(err, data){

                   console.log(err);
                  console.log(data.Contents);

etc etc

[justinpeake]

Ah, now when I hard code the config vars I get this error:

{ [RequestTimeTooSkewed: The difference between the request time and the current time is too large.]
  message: 'The difference between the request time and the current time is too large.',
  code: 'RequestTimeTooSkewed',
  region: 'us-east-1',
  time: Wed Jun 15 2016 14:44:10 GMT-0400 (EDT),
  requestId: '5E5B30939D73418F',
  extendedRequestId: 'sP5Fx2Jb4DvvM9zOy8HE/joBLMQUCWzUuTO6giBmk7JShPA+lAkskb+t8QRfTD1Ey3We0W/1fhg=',
  cfId: undefined,
  statusCode: 403,
  retryable: false,
  retryDelay: 43.59750817529857 }

[chrisradek]

@justinpeake
If you name your ENV variables as:
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY then the SDK will automatically use those, you won't need to store them and update your config yourself.

The clock skew error is interesting. It happens when the date/time of your environment is too far off from the date/time of the service you're calling. You can sometimes see that in browser or mobile environments where an internet connection is dropped and then requests resume once the connection is back, I wouldn't expect that from a node app. You can configure the SDK to attempt to correct the clock skew problems by setting the correctClockSkew config option to true:
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Config.html#correctClockSkew-property

[justinpeake]

Thanks. I commented out the line which updates config... same problem (although good to know).

I'll try to deal with clockSkew:
Can you tell me where I should put the aws.config.correctClockSkew(true) in the code? Should it just be in the global scope?

[chrisradek]

It makes sense to add it to the global scope since the clock skew issue should affect all services if it affects one.

You can update it by doing something like this:

aws.config.update({correctClockSkew: true});

[justinpeake]

That did it. Thank you much! I'm gonna research the clockSkew thing and try and understand it better.