S3 - copy files between buckets from different providers.

[MarcinS-Qlos]

Hello, I have problem with copying files from DIgitalOcean S3 bucket to AWS S3 bucket.

I'm successfully using this syntax while syncing DO S3 bucket with OVH S3 bucket:
aws s3 sync --profile digitalocean --endpoint https://fra1.digitaloceanspaces.com s3://[do bucket]/ --profile ovh --endpoint https://s3.waw.cloud.ovh.net s3://[ovh bucket]/

But when i try the same with AWS it doesn't work:
aws s3 sync --profile digitalocean --endpoint https://fra1.digitaloceanspaces.com s3://[do bucket]/ --profile amazon --endpoint https://s3.eu-central-1.amazonaws.com s3://[AWS bucket]/

I tried also without providing endpoint but result is the same, AWS Cli only tries to connect to one provider and searches for both buckets there.

Is there any, simply, way to sync buckets?
Simplest way would be stick with OVH but S3 lacks functionality to use versioning and automatically delete old files not present on source.

DO-AWS.txt
DO-OVH.txt

[aBurmeseDev]

Hi @MarcinS-Qlos, thanks for reaching out.

I noticed from your logs that you're using older version of AWS CLI. Have you considered updating it? Here's reference guide on updating CLI version: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Unfortunately, we don't document support for third-party services like DigitalOcean and OVH. I would suggest reaching out to them directly via DigitalOcean and OVH.

[MarcinS-Qlos]

I've updated AWS CLI, almost nothing changed
DO-AWS 2 .txt

But problem it's not about OVH or DO. It works fine between them. The problem occurs when I try to copy or sync fromy any S3 to Amazon. I think this issue is related to how AWS CLI handles connections when Amazon S3 is used.

[aBurmeseDev]

Thanks for sharing the debug logs. Here's what I found:

An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

There are a few possible causes for this error:

• The bucket name was entered incorrectly
• The bucket doesn't exist or IAM policy doesn't have permission for the role
• You might be in the incorrect region

Could you try listing your buckets with aws s3 ls command or look for the bucket in AWS Console? If you can't find the bucket or the bucket name doesn't show up, it's likely because you are looking in the wrong region.

Hope it helps!

[MarcinS-Qlos]

Bucket named like DO bucket doesn't exist on Amazon. Program is connecting only to Amazon instead of both servers.

I can list both buckets separately

root@manage:~# aws s3 ls  --profile digitalocean --endpoint https://fra1.digitaloceanspaces.com s3://[DO Bucket]/
                           PRE 1/
                           PRE 100000006/
                           PRE 100000034/
                           PRE 100000064/
                           PRE 100000069/
                           PRE 1187/
                           PRE 1436/
                           PRE 1556/
                           PRE 1558/
                           PRE 1574/
                           PRE 1805/
                           PRE 1931/
                           PRE 1948/
                           PRE 1953/
                           PRE 2/
                           PRE 2156/
                           PRE 2167/
                           PRE 2176/
                           PRE 2460/
                           PRE 2575/
                           PRE 2578/
                           PRE 2621/
                           PRE 2728/
                           PRE 2797/
                           PRE 2864/
                           PRE 2875/
                           PRE 3/
                           PRE 3139/
                           PRE 3181/

and amazon - it's empty so I use debug mode to make sure it works:

root@manage:~# aws s3 ls   --profile amazon --endpoint https://s3.eu-central-1.amazonaws.com s3://[AWS Bucket]/ --debug
2022-10-07 08:01:21,486 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.8.1 Python/3.9.11 Linux/5.4.0-90-generic exe/x86_64.ubuntu.20
2022-10-07 08:01:21,486 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--profile', 'amazon', '--endpoint', 'https://s3.eu-central-1.amazonaws.com', 's3://[AWS Bucket]/', '--debug']
2022-10-07 08:01:21,500 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f4b1a9f1a60>
2022-10-07 08:01:21,500 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f4b1ab498b0>
2022-10-07 08:01:21,500 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2022-10-07 08:01:21,500 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f4b1abec9d0>
2022-10-07 08:01:21,500 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f4b1abf1af0>
2022-10-07 08:01:21,501 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f4b1a984430>
2022-10-07 08:01:21,501 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f4b1ab174c0>
2022-10-07 08:01:21,501 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2022-10-07 08:01:21,501 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7f4b1a97e670>
2022-10-07 08:01:21,501 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.8.1/dist/awscli/data/cli.json
2022-10-07 08:01:21,504 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7f4b1aa4b430>
2022-10-07 08:01:21,504 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7f4b1aa4bf70>
2022-10-07 08:01:21,504 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7f4b1aa4bee0>
2022-10-07 08:01:21,504 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7f4b1aa4d0d0>
2022-10-07 08:01:21,504 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7f4b1aa4d040>
2022-10-07 08:01:21,504 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7f4b1a91ee40>
2022-10-07 08:01:21,504 - MainThread - botocore.session - DEBUG - Setting config variable for profile to 'amazon'
2022-10-07 08:01:21,505 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.8.1 Python/3.9.11 Linux/5.4.0-90-generic exe/x86_64.ubuntu.20 prompt/off
2022-10-07 08:01:21,505 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'ls', '--profile', 'amazon', '--endpoint', 'https://s3.eu-central-1.amazonaws.com', 's3://[AWS Bucket]/', '--debug']
2022-10-07 08:01:21,505 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7f4b1a9740d0>
2022-10-07 08:01:21,505 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f4b1b432940>
2022-10-07 08:01:21,506 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7f4b1a95b8b0>
2022-10-07 08:01:21,506 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7f4b1b42ed30>
2022-10-07 08:01:21,506 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f4b1b399790>
2022-10-07 08:01:21,507 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2022-10-07 08:01:21,508 - MainThread - botocore.credentials - DEBUG - Skipping environment variable credential check because profile name was explicitly set.
2022-10-07 08:01:21,509 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f4b1ab173a0>
2022-10-07 08:01:21,509 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7f4b1ab46790>
2022-10-07 08:01:21,509 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x7f4b1a97e670>
2022-10-07 08:01:21,510 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,510 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3_ls: calling handler <function add_waiters at 0x7f4b1a97e670>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.paths: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.anonymous: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.human-readable: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f4b1b3e9100>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.summarize: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.ls: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f4b1b3e9100>
2022-10-07 08:01:21,511 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.ls.request-payer: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f4b1a0f0160>
2022-10-07 08:01:21,512 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2022-10-07 08:01:21,512 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2022-10-07 08:01:21,512 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2022-10-07 08:01:21,512 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2022-10-07 08:01:21,513 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
2022-10-07 08:01:21,514 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.8.1/dist/awscli/botocore/data/endpoints.json
2022-10-07 08:01:21,531 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f4b1cfb0310>
2022-10-07 08:01:21,547 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.8.1/dist/awscli/botocore/data/s3/2006-03-01/service-2.json
2022-10-07 08:01:21,559 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7f4b1d01c280>
2022-10-07 08:01:21,559 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7f4b1d01c040>
2022-10-07 08:01:21,562 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)
2022-10-07 08:01:21,578 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.8.1/dist/awscli/botocore/data/s3/2006-03-01/paginators-1.json
2022-10-07 08:01:21,578 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.s3.ListObjectsV2: calling handler <function base64_decode_input_blobs at 0x7f4b1a968040>
2022-10-07 08:01:21,579 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function set_list_objects_encoding_type_url at 0x7f4b1cd4b8b0>
2022-10-07 08:01:21,579 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function validate_bucket_name at 0x7f4b1cd475e0>
2022-10-07 08:01:21,579 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f4b19a24b80>>
2022-10-07 08:01:21,579 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f4b19a24c10>>
2022-10-07 08:01:21,579 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.s3.ListObjectsV2: calling handler <function generate_idempotent_uuid at 0x7f4b1cd47430>
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function add_expect_header at 0x7f4b1cd47940>
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f4b19a24b80>>
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event before-call.s3.ListObjectsV2: calling handler <function inject_api_version_header_if_needed at 0x7f4b1cd4bca0>
2022-10-07 08:01:21,580 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ListObjectsV2) with params: {'url_path': '/[AWS Bucket]?list-type=2', 'query_string': {'prefix': '', 'delimiter': '/', 'encoding-type': 'url'}, 'method': 'GET', 'headers': {'User-Agent': 'aws-cli/2.8.1 Python/3.9.11 Linux/5.4.0-90-generic exe/x86_64.ubuntu.20 prompt/off command/s3.ls'}, 'body': b'', 'url': 'https://s3.eu-central-1.amazonaws.com/[AWS Bucket]?list-type=2&prefix=&delimiter=%2F&encoding-type=url', 'context': {'client_region': 'eu-central-1', 'client_config': <botocore.config.Config object at 0x7f4b19a24070>, 'has_streaming_input': False, 'auth_type': None, 'encoding_type_auto_set': True, 'signing': {'bucket': '[AWS Bucket]'}}}
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f4b19a24040>>
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7f4b19a24c70>>
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event choose-signer.s3.ListObjectsV2: calling handler <function set_operation_specific_signer at 0x7f4b1cd47310>
2022-10-07 08:01:21,580 - MainThread - botocore.hooks - DEBUG - Event before-sign.s3.ListObjectsV2: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f4b19a24c70>>
2022-10-07 08:01:21,581 - MainThread - botocore.utils - DEBUG - Using S3 path style addressing.
2022-10-07 08:01:21,581 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-10-07 08:01:21,581 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
GET
/[AWS Bucket]
delimiter=%2F&encoding-type=url&list-type=2&prefix=
host:s3.eu-central-1.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20221007T080121Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-10-07 08:01:21,581 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20221007T080121Z
20221007/eu-central-1/s3/aws4_request
380992f2d64bd8836398bc2bfe7ffe8517835aa9bbe40e8755b038fa3f04cc07
2022-10-07 08:01:21,581 - MainThread - botocore.auth - DEBUG - Signature:
6d9fdf395ee95bc57cfb181c25994a596b33fae01b8d5907323387a418946cae
2022-10-07 08:01:21,582 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://s3.eu-central-1.amazonaws.com/[AWS Bucket]?list-type=2&prefix=&delimiter=%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.8.1 Python/3.9.11 Linux/5.4.0-90-generic exe/x86_64.ubuntu.20 prompt/off command/s3.ls', 'X-Amz-Date': b'20221007T080121Z', 'X-Amz-Content-SHA256': b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIA4WT5JN7TH3Y66WPJ/20221007/eu-central-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=6d9fdf395ee95bc57cfb181c25994a596b33fae01b8d5907323387a418946cae'}>
2022-10-07 08:01:21,582 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.8.1/dist/awscli/botocore/cacert.pem
2022-10-07 08:01:21,583 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): s3.eu-central-1.amazonaws.com:443
2022-10-07 08:01:21,745 - MainThread - urllib3.connectionpool - DEBUG - https://s3.eu-central-1.amazonaws.com:443 "GET /[AWS Bucket]?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1" 200 None
2022-10-07 08:01:21,745 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amz-id-2': 'kZmPFgNBmmn2lpgnvuEgbDJoQ4oZy4mURDYAjdl8K5kogdHx2Qvu1OcRset62XTGSj54x4wsJ0k=', 'x-amz-request-id': '758RRVH8DCDCY9DK', 'Date': 'Fri, 07 Oct 2022 08:01:22 GMT', 'x-amz-bucket-region': 'eu-central-1', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}
2022-10-07 08:01:21,746 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>[AWS Bucket]</Name><Prefix></Prefix><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><EncodingType>url</EncodingType><IsTruncated>false</IsTruncated></ListBucketResult>'
2022-10-07 08:01:21,746 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f4b19a24af0>>
2022-10-07 08:01:21,746 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2022-10-07 08:01:21,746 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f4b19a24b80>>
2022-10-07 08:01:21,746 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x7f4b1cd4b9d0>
2022-10-07 08:01:21,747 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x7f4b1a9f1ca0>
2022-10-07 08:01:21,747 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f4b19a246a0>>

[JD-ISWT]

Hello @MarcinS-Qlos I read that you succeed to sync from DO to OVH
I am wondering on how you did this

I tried to do the same from OVH to Scaleway, but seems the profile arg is read only once, so the source in never read

Do you have a specific setting for your DO bucket? Maybe it is a public bucket?

[MarcinS-Qlos]

Hello @JD-ISWT after further testing I figured out that my client made a bucket at OVH with same name as the source bucket on DO. And he also used external software to sync those buckets so all our tests showed that buckets were syncing properly between DO and OVH.
So as far as I know AWS CLI can't connect to 2 buckets from different providers. I bypassed this issue by s3fs destination bucket and syncing source bucket and 'local' folder. It's not perfect but it's enough for me.

[JD-ISWT]

@MarcinS-Qlos Thks for confirmation
I'm going to test rclone on my side.