Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preserving timestamps of files via a flag? #532

Open
Ahuge opened this issue Nov 1, 2022 · 2 comments
Open

Preserving timestamps of files via a flag? #532

Ahuge opened this issue Nov 1, 2022 · 2 comments
Assignees
Labels

Comments

@Ahuge
Copy link

Ahuge commented Nov 1, 2022

Hi, I have a use case where I am using s5cmd as a "stash" for user's uncommitted files from Perforce and this has been working great in terms of speed, however Perforce takes the timestamps on the files in account when determining if the user need to refresh their working space.

I am looking to see if we can include a flag that will record timestamps of files in the s3 metadata and then re-apply them during download. I'll see if I can get a PR up to help with this.

Thanks!
-Alex

@ajw725
Copy link

ajw725 commented Jun 9, 2023

this is also critical for the sync command that was introduced in 2.0.0. aws s3 sync does preserve the timestamps of the files, so if you sync a directory from S3 to local and then turn right back around and sync it back to S3, it's a no-op because the sizes and timestamps haven't changed. if you try to do that with s5cmd, it copies every single file back up to S3, because it changes the last modified timestamps when it syncs the directory down from S3.

(the actual use case here involves changing/adding some files between the download and the upload.)

@bounlu
Copy link

bounlu commented Aug 3, 2023

Yes, I also agree that s5cmd should preserve timestamp as it would be in case of aws s3. I got into below issue today trying to figure out why my files look older than expected:

$ aws s3 ls s3://xxx
2023-08-02	14:16:07	50618045579	abc_1_val_1.fq.gz
2023-08-02	14:16:07	45907164979	abc_2_val_2.fq.gz
2023-08-02	16:25:17	74599091180	abc_1_val_1.fq.gz
2023-08-02	16:25:17	67202267370	abc_2_val_2.fq.gz
2023-08-02	16:05:07	68417358438	abc_1_val_1.fq.gz
2023-08-02	16:05:07	61942134475	abc_2_val_2.fq.gz
2023-08-02	15:06:47	60423742857	abc_1_val_1.fq.gz
2023-08-02	15:06:47	54783772662	abc_2_val_2.fq.gz
2023-08-02	14:44:29	51372815911	abc_1_val_1.fq.gz
2023-08-02	14:44:29	47378846706	abc_2_val_2.fq.gz

$ s5cmd ls s3://xxx
2023/08/02	06:16:07	50618045579	abc_1_val_1.fq.gz
2023/08/02	06:16:07	45907164979	abc_2_val_2.fq.gz
2023/08/02	08:25:17	74599091180	abc_1_val_1.fq.gz
2023/08/02	08:25:17	67202267370	abc_2_val_2.fq.gz
2023/08/02	08:05:07	68417358438	abc_1_val_1.fq.gz
2023/08/02	08:05:07	61942134475	abc_2_val_2.fq.gz
2023/08/02	07:06:47	60423742857	abc_1_val_1.fq.gz
2023/08/02	07:06:47	54783772662	abc_2_val_2.fq.gz
2023/08/02	06:44:29	51372815911	abc_1_val_1.fq.gz
2023/08/02	06:44:29	47378846706	abc_2_val_2.fq.gz

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Progress
Development

No branches or pull requests

5 participants