s3 ls --recursive fails after long time

[opennomad]

running on EC2 with aws-cli/1.7.4 Python/2.7.6 Linux/3.13.0-44-generic I'm trying to list all objects in a S3 bucket with --recursive. There are some 12 million objects and I've only managed to get about half of them listed before I get the error:

"A client error (InvalidAccessKeyId) occurred when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records."

So, the key definitely works since i'm getting a lot of the listing, but after a long time (between 1 and 2 hours) it fails with the above.

[jamesls]

How are you configuring credentials? Are you launching the instance with an IAM role? Or did you run aws configure and manually enter your credentials? If you're using an IAM role it might be an issue with the way that we're refreshing credentials

[opennomad]

I'm using an IAM role and it's configured via .aws/credentials etc
On Jan 29, 2015 3:04 PM, "James Saryerwinnie" notifications@github.com
wrote:

How are you configuring credentials? Are you launching the instance with
an IAM role? Or did you run aws configure and manually enter your
credentials? If you're using an IAM role it might be an issue with the way
that we're refreshing credentials

—
Reply to this email directly or view it on GitHub
#1118 (comment).

[jamesls]

Are you saying that you have your keys actually in the ~/.aws/credentials file? For example:

[default]
aws_access_key_id=foo
aws_secret_access_key=bar

[opennomad]

Sorry, I wasn't clear. Yes. That's how I have my keys configured. I also
have a .aws/config where my default region is set.

On Thu, Jan 29, 2015 at 3:13 PM, James Saryerwinnie <
notifications@github.com> wrote:

Are you saying that you have your keys actually in the ~/.aws/credentials
file? For example:

[default]
aws_access_key_id=foo
aws_secret_access_key=bar

—
Reply to this email directly or view it on GitHub
#1118 (comment).

[opennomad]

OK, so after numerous failures and finally opening an issue the ls finally completed without error. Took about 2.5 hours. I've done some more digging and found that there possibly were updates to the cloudformation template during my earlier runs. Would that possible cause an interuption or issue with the keys or did I just finally get lucky?

[jamesls]

I wouldn't expect cloudformation template changes to affect the aws s3 ls command. Not sure what the cause could be. If we were getting credentials from the IAM instance metadata service than it might have been an issue with how we refresh credentials. But if we're just pulling from ~/.aws/credentials and the credentials are not changing over time, I'm not sure how we could periodically get a "The AWS Access Key Id you provided does not exist in our record" error message.

I'll try setting up a test environment with a large number of objects and see if I can repo the issue.

[jamesls]

I've ran a some tests several times and I'm not able to see this issue.

If the cloudformation template doing anything with IAM users or policies? It is possible that if those resources (or any related resources) were changing they could have affected the aws s3 ls command. That actually seems like the most likely issue.

Let me know if you're still seeing any issues.

[opennomad]

So, the cloudformation templates didn't explicitly change any of the IAM policies. It is possible that things were added to it and such, but that's a bit of a black box. I took a look around to see if there are any hints as to SGs changing, but couldn't find such a smoking gun.

At any rate, I appreciate your effort in looking into this. I have managed to get the full listing and haven't had the error recur. I'll go ahead an close this and tentatively blame it on the stack-update calls.