Feat-large-file-s3#17
Conversation
| $data = \file_get_contents($source); | ||
| $this->headers['date'] = \gmdate('D, d M Y H:i:s T'); | ||
| $this->headers['content-type'] = \mime_content_type($source); | ||
| $this->headers['content-md5'] = \base64_encode(md5($data, true)); //TODO whould this work well with big file? can we skip it? |
There was a problem hiding this comment.
//TODO whould this work well with big file? can we skip it?
Found this in the comments of md5() docs:
If you want to hash a large amount of data you can use the hash_init/hash_update/hash_final functions.
This allows you to hash chunks/parts/incremental or whatever you like to call it.
There was a problem hiding this comment.
Thanks, but I think we can skip this for now, as we will be using small chunks for uploading. Any files above 5 MB will be chunked!
| // More info on how AWS calculates ETag for multipart upload here | ||
| // https://savjee.be/2015/10/Verifying-Amazon-S3-multi-part-uploads-with-ETag-hash/ | ||
| // TODO | ||
| // $this->assertEquals(\md5_file($source), $this->object->getFileHash($dest)); |
There was a problem hiding this comment.
Should we address this TODO before merging?
There was a problem hiding this comment.
I don't think we can, and may be we don't need to. For every chunk upload we are passing the md5 for the chunk, which is validated otherwise the part upload fails. And if all parts are uploaded correctly we can validate that, the overall data is uploaded correctly right?
Also, don't know how to address this, AWS calculates MD5 differently for multipart upload, hashes the hash of individual parts or something. So the original file hash will not match. If you have an idea do share.
| $uri = ($path !== '') ? '/' . \str_replace('%2F', '/', \rawurlencode($path)) : '/'; | ||
| if($length !== null) { | ||
| $end = $offset + $length - 1; | ||
| $this->headers['range'] = "bytes=$offset-$end"; |
There was a problem hiding this comment.
Is this the intended string value for the range header?
['range' => 'bytes=$offset-$end']
There was a problem hiding this comment.
Yes, it is. If reading a chunk the range will provide which part of the file was read.
| $this->headers['content-type'] = \mime_content_type($source); | ||
| $this->headers['content-md5'] = \base64_encode(md5($data, true)); //TODO whould this work well with big file? can we skip it? | ||
| $this->amzHeaders['x-amz-content-sha256'] = \hash('sha256', $data); | ||
| unset($this->amzHeaders['x-amz-acl']); |
There was a problem hiding this comment.
Updated answer in comment.
3 participants
Large file upload in S3 using multipart upload
Open Question
abortMultipartUpload, if user doesn't want to complete or while clearing incomplete multipart upload, user needs to callabortMultipartUploadto remove the parts that are already uploaded otherwise they will be charged for those storage until they abort?