ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid #297

quantonganh · 2014-03-20T13:54:39Z

s3cmd was installed from epel-testing repo by running:

yum --enablerepo epel-testing install s3cmd

Then I invoked the configuration tool with s3cmd --configure but I got this error:

Test access with supplied credentials? [Y/n] 
Please wait, attempting to list all buckets...
ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid.

Invoked as: /usr/bin/s3cmd --configureProblem: AttributeError: 'S3Error' object has no attribute 'find'
S3cmd:   1.5.0-beta1
python:   2.6.8 (unknown, Mar 14 2013, 09:31:22) 
[GCC 4.6.2 20111027 (Red Hat 4.6.2-2)]

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 2323, in <module>
    main()
  File "/usr/bin/s3cmd", line 2221, in main
    run_configure(options.config, args)
  File "/usr/bin/s3cmd", line 1704, in run_configure
    if e.find('403') != -1:
AttributeError: 'S3Error' object has no attribute 'find'

I'm sure the keys are correct.

Do you have any idea about this?

PS: I also tried with the latest version from github, nothing change.

The text was updated successfully, but these errors were encountered:

mdomsch · 2014-03-20T17:52:19Z

Does the token have sufficient permissions to allow listing all buckets?
that's usually what this error means.

On Thu, Mar 20, 2014 at 8:54 AM, quantonganh notifications@github.comwrote:

s3cmd was installed from epel-testing repo by running:

yum --enablerepo epel-testing install s3cmd

Then I invoked the configuration tool with s3cmd --configure but I got
this error:

Test access with supplied credentials? [Y/n]
Please wait, attempting to list all buckets...
ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid.

Invoked as: /usr/bin/s3cmd --configureProblem: AttributeError: 'S3Error' object has no attribute 'find'
S3cmd: 1.5.0-beta1
python: 2.6.8 (unknown, Mar 14 2013, 09:31:22)
[GCC 4.6.2 20111027 (Red Hat 4.6.2-2)]

Traceback (most recent call last):
File "/usr/bin/s3cmd", line 2323, in
main()
File "/usr/bin/s3cmd", line 2221, in main
run_configure(options.config, args)
File "/usr/bin/s3cmd", line 1704, in run_configure
if e.find('403') != -1:
AttributeError: 'S3Error' object has no attribute 'find'

I'm sure the keys are correct.

Do you have any idea about this?

PS: I also tried with the latest version from github, nothing change.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297
.

mdomsch · 2014-03-20T17:54:48Z

The token must have ListAllMyBuckets permission.

On Thu, Mar 20, 2014 at 12:52 PM, Matt Domsch matt@domsch.com wrote:

Does the token have sufficient permissions to allow listing all buckets?
that's usually what this error means.

On Thu, Mar 20, 2014 at 8:54 AM, quantonganh notifications@github.comwrote:

s3cmd was installed from epel-testing repo by running:

yum --enablerepo epel-testing install s3cmd

Then I invoked the configuration tool with s3cmd --configure but I got
this error:

Test access with supplied credentials? [Y/n]
Please wait, attempting to list all buckets...
ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid.

Invoked as: /usr/bin/s3cmd --configureProblem: AttributeError: 'S3Error' object has no attribute 'find'
S3cmd: 1.5.0-beta1
python: 2.6.8 (unknown, Mar 14 2013, 09:31:22)
[GCC 4.6.2 20111027 (Red Hat 4.6.2-2)]

Traceback (most recent call last):
File "/usr/bin/s3cmd", line 2323, in
main()
File "/usr/bin/s3cmd", line 2221, in main
run_configure(options.config, args)
File "/usr/bin/s3cmd", line 1704, in run_configure
if e.find('403') != -1:
AttributeError: 'S3Error' object has no attribute 'find'

I'm sure the keys are correct.

Do you have any idea about this?

PS: I also tried with the latest version from github, nothing change.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297
.

quantonganh · 2014-03-21T03:10:22Z

@mdomsch I'm an IAM admin. My Group Policies:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

Moreover, if I login to the S3 Management Console using Sign-In Credentials, I can see the list of all buckets. The strange thing is I didn't get this problem on another server with the same token.

For e.g: here's the output when running s3cmd ls on my workstation (with the same token):

$ s3cmd ls
2013-11-06 02:48  s3://foo-1
2013-11-06 02:50  s3://foo-2
...

What the hell is going on here?

mdomsch · 2014-03-21T04:30:28Z

Can you run with --debug on the latest github upstream on the failing
system? I'd love to see what's being sent and returned in detail, which
--debug will provide.

Thanks,
Matt

On Thu, Mar 20, 2014 at 10:10 PM, quantonganh notifications@github.comwrote:

@mdomsch https://github.com/mdomsch I'm an IAM admin. My Group Policies:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "",
"Resource": ""
}
]
}

Moreover, if I login to the S3 Management Console using Sign-In
Credentials, I can see the list of all buckets. The strange thing is I
didn't get this problem on another server with the same token.

For e.g: here's the output when running s3cmd ls on my workstation (with
the same token):

$ s3cmd ls
2013-11-06 02:48 s3://foo-1
2013-11-06 02:50 s3://foo-2
...

What the hell is going on here?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297#issuecomment-38245885
.

quantonganh · 2014-03-21T07:47:48Z

@mdomsch Found some clues: don't know why s3cmd added x-amz-security-token to the SignHeaders automatically.

On the worked system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:07:18 +0000\n/'

On the failed system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:40:56 +0000\nx-amz-security-token:AQoDYXdzENH...\n/'

Looks like this is the reason why I got a 400 (Bad Request) response.

Where did it come from?

mdomsch · 2014-03-21T14:07:45Z

    if len(self.s3.config.access_token)>0:
        self.s3.config.role_refresh()
        self.headers['x-amz-security-token']=self.s3.config.access_token

came from

commit dc590d6
Author: David Kohen kohen.d@gmail.com
Date: Wed Feb 13 15:57:11 2013 +0200

Add support for IAM roles and temp tokens

Including:
Refresh temp tokens on every request.
Add support for roles in S3, CloudFront and SimpleDB.
Add support for AWS_CREDENTIALS_FILE env variable and the file it

references

On Fri, Mar 21, 2014 at 2:47 AM, quantonganh notifications@github.comwrote:

@mdomsch https://github.com/mdomsch Found some clues: don't know why
s3cmd added x-amz-security-token to the SignHeaders automatically.

On the worked system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:07:18 +0000\n/'

On the failed system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:40:56 +0000\nx-amz-security-token:AQoDYXdzENH...\n/'

Looks like this is the reason why I got a 400 response.

Where did it come from?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297#issuecomment-38254767
.

quantonganh · 2014-03-21T14:21:19Z

@mdomsch: I also have found this via Google: http://www.greenhills.co.uk/2012/12/25/s3cmd-with-iam-roles.html

So, I know that it was taken from the metadata:

# wget -O - -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole'
{
  "Code" : "Success",
  "LastUpdated" : "2014-03-21T12:45:27Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "foo",
  "SecretAccessKey" : "bar",
  "Token" : "AQoDYXdzENH....",
  "Expiration" : "2014-03-21T19:18:02Z"
}

But my question still stands: why does sometimes s3cmd add x-amz-security-token to the header when running --configure (I am not using --add-header)?

mdomsch · 2014-03-21T17:54:37Z

Is s3cmd run on a system with an IAM role defined? Is the access token
defined in the configuration or environment?
This has been running on production servers for over a year.
On Mar 21, 2014 4:07 PM, "Matt Domsch" matt@domsch.com wrote:

    if len(self.s3.config.access_token)>0:
        self.s3.config.role_refresh()
self.headers['x-amz-security-token']=self.s3.config.access_token

came from

commit dc590d6
Author: David Kohen kohen.d@gmail.com
Date: Wed Feb 13 15:57:11 2013 +0200
Add support for IAM roles and temp tokens

Including:
Refresh temp tokens on every request.
Add support for roles in S3, CloudFront and SimpleDB.
Add support for AWS_CREDENTIALS_FILE env variable and the file it
references

On Fri, Mar 21, 2014 at 2:47 AM, quantonganh notifications@github.comwrote:

@mdomsch https://github.com/mdomsch Found some clues: don't know why
s3cmd added x-amz-security-token to the SignHeaders automatically.

On the worked system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:07:18 +0000\n/'

On the failed system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:40:56 +0000\nx-amz-security-token:AQoDYXdzENH...\n/'

Looks like this is the reason why I got a 400 response.

Where did it come from?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297#issuecomment-38254767
.

quantonganh · 2014-03-22T01:02:27Z

Is s3cmd run on a system with an IAM role defined?

Yes, both of the worked and failed system are associated with the same IAM role.

Is the access token defined in the configuration or environment?

Sorry, I'm not sure I understand your question. I'm going to run s3cmd --configure at the first time to generate the config file ~/.s3cfg. Don't know why it takes the Token from the metadata and appended to the header.

By "environment", do you mean AWS_TOKEN? If so, there is nothing related to this variables. Here're all the AWS variables on the failed system (same as worked system):

declare -x AWS_ACCESS_KEY="foo"
declare -x AWS_ACCESS_KEY_ID=""
declare -x AWS_AUTO_SCALING_HOME="/opt/aws/apitools/as"
declare -x AWS_CLOUDWATCH_HOME="/opt/aws/apitools/mon"
declare -x AWS_ELB_HOME="/opt/aws/apitools/elb"
declare -x AWS_IAM_HOME="/opt/aws/apitools/iam"
declare -x AWS_PATH="/opt/aws"
declare -x AWS_RDS_HOME="/opt/aws/apitools/rds"
declare -x AWS_SECRET_KEY="bar"

mdomsch · 2014-03-22T04:08:14Z

I just pushed a change to master to display the actual error (and not crap
out) returned from S3, with the S3 error message, when --configure fails.
Please pull te master branch and try again.

On Fri, Mar 21, 2014 at 8:02 PM, quantonganh notifications@github.comwrote:

Is s3cmd run on a system with an IAM role defined?

Yes, both of the worked and failed system are associated with the same IAM
role.

Is the access token defined in the configuration or environment?

Sorry, I'm not sure I understand your question. I'm going to run s3cmd
--configure at the first time to generate the config file ~/.s3cfg. Don't
know why it takes the Token from the metadata and appended to the header.

What do you mean by "environment"?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297#issuecomment-38338765
.

mdomsch · 2014-03-22T04:16:25Z

environment, meaning printenv from the shell.

I just tried upstream master branch which has a fix for the find() error
noted above now, running:
rm ~/.s3cfg
./s3cmd --configure

and entering in a newly created set of API keys. This worked fine. If I
set the permissions on these keys to disallow ListAllMyBuckets, it errors
out with an appropriate message. If I then clear the permissions and set
only ListAllMyBuckets, --configure succeeds (but the key can't do more than
that).

On Fri, Mar 21, 2014 at 8:02 PM, quantonganh notifications@github.comwrote:

Is s3cmd run on a system with an IAM role defined?

Yes, both of the worked and failed system are associated with the same IAM
role.

Is the access token defined in the configuration or environment?

Sorry, I'm not sure I understand your question. I'm going to run s3cmd
--configure at the first time to generate the config file ~/.s3cfg. Don't
know why it takes the Token from the metadata and appended to the header.

What do you mean by "environment"?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/297#issuecomment-38338765
.

quantonganh · 2014-03-22T04:53:29Z

@mdomsch: This is better because it allows users to save settings or not.

As you can guess, if I choose to save the settings, then open ~/.s3cfg file, I will see something like this:

access_token = AQoDYXdzEOX//////////...

By removing this line, s3cmd will work fine.

What I don't understand is why s3cmd sometimes take the access token to used as a temporary credential, sometimes it doesn't? I don't care about this IAM role, because with the access_key and secret_key, I am an IAM admin. Why the temporary credentials has higher priority than my AWS_ACCESS_KEY and AWS_SECRET_KEY? What can I do to force the s3cmd not to use the token when generating the config file?

PS: I'm re-reading http://aws.amazon.com/iam/faqs/ to make sure that I'm not missing anything important.

russvanderpool · 2015-01-21T20:41:20Z

fwiw - after installing s3cmd, setting up an IAM and attaching an "Amazon S3 Full Access" role - I too encountered the "The provided token is malformed or otherwise invalid" error.... Next, I created an IAM and attached a policy with admin credentials (everything)...

still got error.

So after seeing @mdomsch ask if there was an IAM role attached to the EC2 where the s3cmd is installed, I indeed did have my EC2 setup with an embedded role that had only an EC2 policy attached.

So next, I tried running.
s3cmd --access_key=xxxx --secret_key=xxxxxxxxxxxxx ls
using the IAM with the S3 policy

and of course it works.

So s3cmd takes as a default the credentials of embedded roles in the EC2. I am sure that if you spin up another EC2 with a role that has access to S3 - you will not get this "The provided token is malformed or otherwise invalid" error.

But, please DO NOT stick -access_key=xxxx --secret_key=xxxxxxxxxxxxx parameters into any script using s3cmd. Where ever possible - try to embed roles into EC2 when you fire them up (good security practice)

Anyway - to test - I fired up another EC2 w/out any embedded IAM roles, installed s3cmd, configured - and everything worked as expected using my IAM user that had the S3 policy.

Hopeful s3cmd fix: Allow the IAM credentials set by the "s3cmd --configure" to take precedence over embedded IAM roles in EC2 please.

sokser · 2017-03-03T19:16:20Z

I can confirm that removing access_token = AQoDYXdzEOX//////////... from the .s3cfg file makes the program work, but when it is in the config file, it breaks the program

gjtorikian · 2017-04-28T22:45:14Z

I can also confirm this.

Kukhapprabu-Gladminds · 2017-05-24T07:17:26Z

For me also it worked after removing access_token = AQoDYXdzEOX//////////... from the .s3cfg file.

scervera · 2018-06-05T01:03:02Z

I can confirm that removing the access_token fixes this issue.

knightfoxapps · 2019-01-14T17:14:29Z

had the same error until I used sudo s3cmd --configure

shekargowda11 · 2019-07-22T12:45:37Z

The region name was difference in my case

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid #297

ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid #297

quantonganh commented Mar 20, 2014

mdomsch commented Mar 20, 2014

mdomsch commented Mar 20, 2014

quantonganh commented Mar 21, 2014

mdomsch commented Mar 21, 2014

quantonganh commented Mar 21, 2014

mdomsch commented Mar 21, 2014

quantonganh commented Mar 21, 2014

mdomsch commented Mar 21, 2014

quantonganh commented Mar 22, 2014

mdomsch commented Mar 22, 2014

mdomsch commented Mar 22, 2014

quantonganh commented Mar 22, 2014

russvanderpool commented Jan 21, 2015

sokser commented Mar 3, 2017

gjtorikian commented Apr 28, 2017

Kukhapprabu-Gladminds commented May 24, 2017

scervera commented Jun 5, 2018

knightfoxapps commented Jan 14, 2019

shekargowda11 commented Jul 22, 2019

ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid #297

ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid #297

Comments

quantonganh commented Mar 20, 2014

mdomsch commented Mar 20, 2014

mdomsch commented Mar 20, 2014

quantonganh commented Mar 21, 2014

mdomsch commented Mar 21, 2014

quantonganh commented Mar 21, 2014

mdomsch commented Mar 21, 2014

quantonganh commented Mar 21, 2014

mdomsch commented Mar 21, 2014

quantonganh commented Mar 22, 2014

mdomsch commented Mar 22, 2014

mdomsch commented Mar 22, 2014

quantonganh commented Mar 22, 2014

russvanderpool commented Jan 21, 2015

sokser commented Mar 3, 2017

gjtorikian commented Apr 28, 2017

Kukhapprabu-Gladminds commented May 24, 2017

scervera commented Jun 5, 2018

knightfoxapps commented Jan 14, 2019

shekargowda11 commented Jul 22, 2019