Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gsutil rsync -c alternative for if mtime is changed, then check hash (repeated file upload issue.) #1675

Open
kamarton opened this issue Feb 22, 2023 · 0 comments

Comments

@kamarton
Copy link

Similar to the -c option, a parameter should be added that calculates CRC/MD5 hash in case of differences in mtime, and the file is only moved if there is a difference.

The situation is not clear, but identical files are being synchronized from a NAS server - I assume because there has been a change in the mtime value.

I currently use this:

gsutil -m rsync -r -d "/sourcedir" gs://destination-bucket/

And i see this in google cloud console web interface:

same-file-sync-gs-console

This situation is not constant, but there are files that have been uploaded multiple times, even though it is the same file being synchronized upwards.

Expected

Adding a new parameter that performs a content hash/crc check when the size is the same, but the mtime value is different.

@kamarton kamarton changed the title gsutil rsync -c alternative for if mtime is changed, then check hash gsutil rsync -c alternative for if mtime is changed, then check hash (repeated file upload issue.) Feb 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant