S3 Sync Utility

[jamesrenaud]

Is your feature request related to a problem? Please describe.

One of the great features of the AWS CLI is the S3 sync command (https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html). The PHP SDK for AWS provides similar functionality (https://docs.aws.amazon.com/aws-sdk-php/v2/api/namespace-Aws.S3.Sync.html) which provide the ability to quickly do a diff between a local directory and an S3 path (or vise versa).

The CLI handles a number of excellent edge cases which can be difficult to continually resolve as an implementer:

• Following OS symlinks
• Automatically detecting and setting content-type tags
• Removing files that are no longer present in the local or remote

A few npm packages have sprung up in the Javascript ecosystem (https://www.npmjs.com/search?q=s3%20sync) attempting to address this, however they generally run at a CLI level, are unmaintained, or do not provide an appropriate Nodejs programmatic interface.

It would be advantageous to have this utility built in directly to the AWS SDK minimizing my dependence on third party packages, or needing to write this logic myself across multiple implementations.

Describe the solution you'd like

Add a method to the existing @aws-sdk/client-s3 implementation to perform the sync operation, or add a new @aws-sdk/util-s3-sync that implements this functionality.

Describe alternatives you've considered

To date I've simply spawned a child process and run the CLI command directly, or attempted to implement the additional features of the sync command (like content-type detection, and file diffing). Always wanted this feature in the SDK directly.

Additional context

None.

[jeanbmar]

I support this request, it would be an awesome addition to the SDK

[jeanbmar]

I ended up coding my own module (https://github.com/jeanbmar/s3-sync-client) to tackle the following issues of the existing repo:

• Require AWS CLI to be installed
• Use Etag to perform file comparison (Etag should be considered an opaque field, and should not be used)
• Limit S3 bucket object listing to 1000 objects
• Support syncing bucket with local, but doesn't support syncing local with bucket
• Use outdated dependencies
• Is unmaintained

[ghost]

FYI - there is this new "@aws-sdk/lib-storage" package with the upload abstraction method. It might be a good fit / start for a new sync method.

BTW @jeanbmar, I think you should consider this new update method instead of the PutObjectCommand you're using. This way you would support large files with multipart transfers out of the box.

[jeanbmar]

@fboutin-pmc Good call thank you!

[RanVaknin]

Hi @jamesrenaud,

Thanks for your feature request. At this time we are not able to move forward with the implementation for this.
Feature requests are prioritized based on community engagement, and right now the team's focus is on other issues.

Personally I have no used the community maintained s3 sync client, but it seems fairly well documented and with recent contributions. That might be a possible workaround.

Thanks again!
Ran~

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.