Sync two S3 buckets in two different AWS accounts using AWS Profile + assumeRole to access S3 API?

[frittentheke]

Thanks for creating this great tool!

I attempted to use s3sync with a setup that requires an assumeRole to access the target bucket but apparently this is not supported (yet)?

More particular the setup looks like this:

1. Two AWS account, each containing an S3 bucket:

Account 12345: source-bucket
Account 67890: destination-bucket

2. Source bucket received a policy like:

        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::DESTINATION_ACCOUNT_ID:root"
            },
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::destination-bucket/*",
                "arn:aws:s3:::destination-bucket"
            ]
        }

to allow the destination bucket to list and pull the files.

3. I then used the aws cli tool to verify things work:

export AWS_PROFILE=my.account.test
aws s3 sync s3://source-bucket s3://destination-bucket

and it was syncing the objects quite happily ... just not as fast as I'd like to ;-)

My questons or potential feature request now is to be able to use s3sync in this setup.

[larrabee]

Hello.
Thanks for feedback;).
Are you checked the s3sync with this setup? AWS SDK supports AWS_PROFILE and should read credentials from it.
So you try to run s3sync and report a result.
Please use latest version (2.15). I was build create this release few min ago. Prev release was not contains latest changes for some auth methods.

[frittentheke]

@larrabee thanks for the quick reply. Sorry for not mentioning this in my initial post - of course I did test the setup with s3sync, but it did not work.
I now used the new release, but that also fails:

INFO[0000] Starting sync                                
DEBU[0006] S3 listing failed with error: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors 
DEBU[0006] Pipeline step: ListSource finished           
DEBU[0006] Recv pipeline err: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors 
ERRO[0006] Sync error: pipeline step: 0 (ListSource) failed with error: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors, terminating 
DEBU[0006] Pipeline step: ACLUpdater finished           
DEBU[0006] Pipeline step: LoadObjData finished          
DEBU[0006] Pipeline step: UploadObj finished            
DEBU[0006] Pipeline step: Terminator finished           
DEBU[0006] All pipeline steps finished                  
DEBU[0006] Pipeline terminated                          
INFO[0006] 0 ListSource: Input: 0; Output: 0 (0 obj/sec); Errors: 1 
INFO[0006] 1 LoadObjData: Input: 0; Output: 0 (0 obj/sec); Errors: 0 
INFO[0006] 2 ACLUpdater: Input: 0; Output: 0 (0 obj/sec); Errors: 0 
INFO[0006] 3 UploadObj: Input: 0; Output: 0 (0 obj/sec); Errors: 0 
INFO[0006] 4 Terminator: Input: 0; Output: 0 (0 obj/sec); Errors: 0 
INFO[0006] Duration: 6.27037978s                        
ERRO[0006] Sync Failed  

[larrabee]

Hello again;)
Are you using AWS_PROFILE correctly?
AWS_PROFILE should contain account name from credentials file. File location can be specified by variable AWS_SHARED_CREDENTIALS_FILE.
For example:

export AWS_PROFILE="default"
export AWS_SHARED_CREDENTIALS_FILE="/home/test/aws_creds"
>> cat /home/test/aws_creds
[default]
aws_access_key_id = KEY
aws_secret_access_key = SECRET

Is AWS regions correct? By default it's connecting to us-east-1 region. You can set region by args --sr and --tr.

[frittentheke]

@larrabee yes, I did all that:

$ env | grep AWS

AWS_PROFILE=my.account.test
AWS_SHARED_CREDENTIALS_FILE=/home/myuser/.aws/credentials

The credentials file (/home/myuser/.aws/credentials) looks like this:

[default]
aws_access_key_id = REDACTED
aws_secret_access_key = REDACTED

The difference in my case is that I want to "assume a role". The credentials file contains my access credentials, but then a file ~/.aws/config exists containing:

[profile my.account.test]
role_arn = arn:aws:iam::1234567890:role/myRoleInTheOtherAccount
source_profile = default
region = eu-central-1

So I assume the role of myRoleInTheOtherAccount which actually allows access to the S3 bucket, not the role / account I have when using the aws_access_key_id/aws_secret_access_key.

In short, the aws_secret_access_key authenticate me as a human, the role is the role I have related to the particular account. This is quite common in larger / enterprise usage of AWS accounts.

[larrabee]

I'm commit changes to branch assumeRole_debug.
Can you build and test it?

[frittentheke]

Hey @larrabee thanks for really digging into this! Really appreciate it.
Are we talking about this commit: ded3296 ? What is that intended to do as the line ded3296#diff-44bbcc9d983da65f32aa64529eb190e2R50 is commented out ...

I built s3sync in branch assumeRole_debug with the commit ... but nothing has changed unfortunately.

[larrabee]

Please try new version from branch assumeRole_debug, it's should works (It's working for me).

[frittentheke]

Yeah! It works splendidly.

[larrabee]

Changes merged to master.