New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cloud-AWS: S3 Repository Across-Account fails verification. #17179
Comments
@dadoonet could you take a look at this please? |
or @xuzha :) |
np @clintongormley I will verify this asap :-) |
Thanks @JamesBromberger for reporting the issue. I went through the code, looks like it do have the Please correct me if I'm missing something here. Based on the errors msg, ES complains that it doesn't have the permission to write to the bucket. And I think the feature in #14297 is for specifying a canned in your request when creating a resource. For example, if you set |
I'm seeing this on ES 2.4.0, I get an access denied but the repository is still created and I'm able to snapshot to it successfully. Using the IAM role policy listed in the docs |
I have the same problem on ES 2.4.1, I get an access denied but the repository is still created. |
The issue for me ended up being that the IAM policy needs delete access to test access to the bucket. I had a blanket policy in place that didn't allow delete's on S3 buckets that was causing this issue. Once I removed that, things worked as expected. |
No further feedback on this one. |
Elevating IAM permissions for delete rights resolved this issue for me. |
Elasticsearch version: 2.2.0
JVM version: 1.8.0
OS version: Amazon Linux 2015.09.02
Description of the problem including expected versus actual behavior: Using Cloud-AWS Plugin to define a repository that is across AWS accounts for escrow backps/snapshots
Steps to reproduce:
Provide logs (if relevant):
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object XXX/Dev/Elasticsearch/tests-DJJ3cNL_ScqiNn6P-LQOAw/master.dat-temp","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CBD309137D130093)"}}},"status":500}
Describe the feature:
I am using ES 2.2.0 , and setting up a cross account trust where the receiving bucket (in another AWS account0 requires "bucket-owner-full-control", Using AWS CLi confirms I can put objects with the canned ACl into the prefix I have reserved for this; I can list the objects back, and I can read the content back.
However, when trying to set up a backup location in elasticsearch, it fails the verification. My config is:
{ "type": "s3", "settings": { "bucket": "XXX-logs", "region": "ap-southeast-2", "server_side_encryption": "True", "base_path": "XXX/Dev/Elasticsearch/", "buffer_size": "100mb", "canned_acl": "bucket-owner-full-control" } }
However when submitting this I get:
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3backupsec] path [XXX][Dev][Elasticsearch] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object XXX/Dev/Elasticsearch/tests-DJJ3cNL_ScqiNn6P-LQOAw/master.dat-temp","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: CBD309137D130093)"}}},"status":500}
Which from the S3 logs on the target S3 bucket I see a 403 AccessDenied. The only condition that is required by S3 in my case is the bucket-owner-full-control ACL. It appears that the check in index/snapshots/blobstore/BlobStoreIndexShardRepository.java calling org.elasticsearch.repositories.blobstore.BlobStoreRepository.testBlobPrefix() may not have the canned ACL being added, and thus fails to get set up.
This maybe somewhere around startVerification() in repositories/blobstore/BlobStoreRepository.java.
In my use case, the target bucket is to be write once (no deletes; escrow copy of the backup, so the corresponding "ensure moves are supported" is also likely to fail....
The text was updated successfully, but these errors were encountered: