S3 input error The AWS Access Key Id you provided does not exist in our records

[jordansissel]

(This issue was originally filed by @karasmeitar at elastic/logstash#3193)

---

I'm trying to get some log files from s3 bucket and put it to elasticsearch.
My config file is:
input {
s3 {
bucket => "dist-platform-qa"
prefix => "es_export_data"
credentials =>"/home/dev/logstash-1.4.2/Aws.config"
region_endpoint => "us-east-1"
}
}
output {
elasticsearch {
host => "localhost"
protocol => "http"
port=> "9200"
index=> "all"
}
}

My Aws.config file:

AWS_ACCESS_KEY_ID = "blabla"
AWS_SECRET_ACCESS_KEY = "blabla"

But i'm still getting errors for my Aws access key(he AWS Access Key Id you provided does not exist in our records).
When i check the permissions with s3cmd i can get files from the bucket and everything is ok.
Any idea?

[tuukkamustonen]

Just bumped into this. I first did:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<ec2-instance-profile-name>

And grabbed key + secret from there, inserting them into configs. Now I'm getting the reported "Access Key Id you provided does not exist in our records" message.

With my personal account key+secret, it works.

With separately created IAM user, I get "Access Denied". When I do:

export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY_ID=xxx
aws s3 cp s3://xxxxx .

The IAM user rights work. But not when I insert the key+secret into logstash configs.

Without manually configuring the key+secret, trying to rely on instance-profile directly, I get "Access Denied" error.

[tuukkamustonen]

Nevermind, I was missing a permission in bucket policy. That's why IAM instance-profile and separate IAM user did not work...

Anyway, I can confirm the issue originally reported here.

[rashmip22]

I am getting the same error, I am new to AWS.
Can you please tell me what permission is that?

[tuukkamustonen]

@rashmip22 It's been a while and I'm not using logstash anymore, but you need permission something like:

        "Statement": [{
          "Effect": "Allow",
          "Action": [
            "s3:GetObject"
          ],
          "Resource": [
            "arn:aws:s3:::my-bucket/*",
          ]
        }, {
          "Effect": "Allow",
          "Action": [
            "s3:ListBucket"
          ],
          "Resource": [
            "arn:aws:s3:::my-bucket"
          ],
        }]

[santiagovm]

@tuukkamustonen thanks, that was my problem

[rodrigoacastro]

Thanks a lot!