aws s3 sync to/from credentials

[brackxm]

Add the ability to specify different credentials for the source and the destination when syncing from and to an S3Uri.
For example --source-profile.
aws --source-profile a --profile b s3 sync s3://bucketa s3://bucketb
It is not always possible to get 1 set of credentials for both the source and the destination S3Uri.
For example, because the owner is different, or due to some regulations.

A workaround it to sync to and then from LocalPath, but this has issues: lower performance, needing a big enough local disk, ...

[JordonPhillips]

You can do this by granting cross account access to one of your roles.

[brackxm]

Thank you for the suggestion.
I understand my feature is not needed in theory, but in practice it happens that you can get access to s3 with separate credentials, but without IAM access.

[JordonPhillips]

Copying an object will still require that you have GetObject permissions in your main profile. That said, there is a point to be made that the HeadObject and ListObject calls can be done with a different set of credentials. Marking as a feature request.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[infintesimal]

Hi all, it seems that this issue got dropped in the move from GitHub to UserVoice and back. I have a real life use case scenario where I want to do a cross-account, cross-region s3 copy of a key.
I tried getting this done by writing some custom boto3 code, but either my code is not correct orthe chain of s3/inject.py -> s3transfer/futures.py -> s3transfer/tasks.py -> s3transfer/copies.py -> botocore/client.py

#!/usr/local/bin/python3

import boto3

source_session = boto3.session.Session(
  aws_access_key_id='AKIAXXX',
  aws_secret_access_key='YYYYYY',
  region_name='us-west-2'
)

target_client = boto3.client(
  service_name='sts',
  aws_access_key_id='AKIAZZZZ',
  aws_secret_access_key='WWWWW',
  region_name='us-east-1'
)

assume_role_response = target_client.assume_role(
  RoleArn='arn:aws:iam::AAAAAAAAAAAA:role/magical-role',
  RoleSessionName='target_session'
)

creds = assume_role_response['Credentials']

assumed_role_session = boto3.session.Session(
  aws_access_key_id=creds['AccessKeyId'],
  aws_secret_access_key=creds['SecretAccessKey'],
  aws_session_token=creds['SessionToken'],
  region_name='us-east-1'
)

assumed_role_session_s3_resource = assumed_role_session.resource(
  service_name='s3',
  aws_access_key_id=creds['AccessKeyId'],
  aws_secret_access_key=creds['SecretAccessKey'],
  aws_session_token=creds['SessionToken'],
  region_name='us-east-1'
)

source_session_resource = source_session.resource(
  service_name='s3',
  aws_access_key_id='AKIAXXXXX',
  aws_secret_access_key='YYYYY',
  region_name='us-west-2'
)
target_bucket_name = 'target-bucket'
target_key = 'FOO'

source_bucket_name = 'source-bucket'
source_bucket_key = 'FOO'

source_object_dict = {
  'Bucket': source_bucket_name,
  'Key': source_bucket_key
}

source_bucket = source_session_resource.Bucket(source_bucket_name)
source_object = source_bucket.Object(source_bucket_key)

target_bucket = assumed_role_session_s3_resource.Bucket(target_bucket_name)
target_bucket.Object('foo')
target_object = target_bucket.Object(target_key)

target_object.copy(source_object_dict)

fails with

  File "./foo.py", line 85, in <module>
    target_object.copy(source_object_dict)
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py", line 483, in object_copy
    Config=Config)
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py", line 379, in copy
    return future.result()
  File "/usr/local/lib/python3.7/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.7/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/usr/local/lib/python3.7/site-packages/s3transfer/tasks.py", line 255, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/s3transfer/copies.py", line 110, in _submit
    **head_object_request)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Of course, supply the source creds on aws cli and downloading the file locally works; supplying target creds on aws cli works. It's only when I try to achieve it in one fell swoop as OP tried (equiv of s3 sync) that it breaks down. Currently using a workaround to download source file to worker host; then using assume role upload credentials to upload the local file. I'd love to be in a spot where I don't have to take the extra network and IO hops.

[konstantinj]

also still waiting for this feature.