aws s3 cp streaming to stdout fails silently

[hguturu]

We recently synced one of our s3 buckets to another bucket and were doing checks that the files moved successfully by doing md5 checksums. We noticed that sometimes the aws s3 cp stream truncates without returning an error or non-zero error code. This may only be relevant for large files - some of our files are 100G+.

e.g.
Round 1:

aws s3 cp s3://$BUCKET1/largefile - | md5sum
d1c2e6835a929e32df189ccfddf1d3fe  -
aws s3 cp s3://$BUCKET2/largefile - | md5sum
927958e42554e40a4f3c37fcced4ba22  -

Did our sync not happen properly? Lets try again.
Round 2:

aws s3 cp s3://$BUCKET1/largefile - | md5sum
927958e42554e40a4f3c37fcced4ba22  -
aws s3 cp s3://$BUCKET2/largefile - | md5sum
927958e42554e40a4f3c37fcced4ba22  -

Looks like it did happen successfully. We realized this was a file stream ending early since we have another special file format that has an marker to denote end of file and we weren't detecting that with our downstream tools.

This issue seems some how memory related -- typically when we see these failures aws s3 cp starts using a lot of memory. But, when we repeat the same job for success the memory usage is much lower.

We are using aws s3 --version:
aws-cli/1.16.102 Python/2.7.14 Linux/4.14.94-89.73.amzn2.x86_64 botocore/1.12.92.

[shubhamsre]

If all the work-around is to sync two s3 buckets then why wouldn't you use aws s3 sync

aws s3 sync s3://$BUCKET1/ s3://$BUCKET2/

To Verify:
aws s3 ls --recursive s3://$BUCKET1 --summarize > bucket-contents-bucket1.txt
aws s3 ls --recursive s3://$BUCKET2 --summarize > bucket-contents-bucket2.txt

Compare objects that are in the source and target buckets by using the outputs that are saved to files in the AWS CLI directory. See the following command for detailed output:
aws s3 ls --recursive s3://BUCKETNAME --summarize from aws docs

Further have you tried using aws s3api it is for low level and works when the mime type of the file is unidentifiable or for the multipart uploads.

[hguturu]

The above is not a work around to sync two buckets. We did use aws s3 sync for that. And the main point of the issue wasn't verification, but rather pointing out how aws s3 cp is not consistent on two runs of the same file since it seems to silently (i.e. without error) truncate the stream. The checksum was a simple use case to point out the issue without having to go into details/having concerns that the program the stream is being piped into is the concern.

Also my understanding is aws s3 ls --recursive --summarize only tells you that the contents of the two buckets are byte size same, not if the bytes themselves are the same.

[hguturu]

#2390 and #3551 might be related issues.

[kdaily]

Hi @hguturu, are you still experiencing this behavior? Thanks!

[hguturu]

Hi @kdaily, I haven't had to do this work load in a while. I can try to duplicate it again and see if its an issue since we are now on aws-cli/1.16.300 Python/2.7.16 Linux/4.14.173-137.229.amzn2.x86_64 botocore/1.13.36.

Update: I haven't been able to duplicate it, but its hard for me to say this is no longer an issue since even before this was a hard to reproduce issue that occurred rarely.

I suspect its related to how the cli tools take up a lot of memory sometimes (perhaps due to network congestion and buffering? or the issues in tickets tagged above?), so if nothing has been done to address them, then its possible the issue is still there.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.

[hguturu]

Hello, I no longer have access to the same environment to reproduce this issue. Closing seems reasonable since the CLI tools are on a new major version. I will re-open if I come across this issue again.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.