Large file upload

[linuxadmin]

I get the following error when uploading a 20gb file. It works when I upload a 10GB file.

DIE outside EVAL block [0]
Call stack:
main::process_forks(./mtglacier.pl:83)
ChildWorker::process(./mtglacier.pl:175)
GlacierRequest::finish_multipart_upload(ChildWorker.pm:54)
main::ANON(GlacierRequest.pm:284)
Fatal Error: 0 Can't call method "header" on an undefined value at GlacierRequest.pm line 284, line 961.
DIE outside EVAL block [0]
Call stack:
main::process_forks(./mtglacier.pl:83)
ParentWorker::process_task(./mtglacier.pl:213)
ParentWorker::wait_worker(ParentWorker.pm:29)
main::ANON(ParentWorker.pm:53)
Fatal Error: 0 Unexpeced EOF in Pipe at ParentWorker.pm line 53, line 960.

[vsespb]

Hello. I am currently on leave, so can be slow in responding. Are you sure yo're using latest version of the software ? (tag v0.7beta https://github.com/vsespb/mt-aws-glacier/zipball/v0.7beta == current master branch).

Will try to reproduce with 20Gb when back from a vacation (next week).

Thanks.

[linuxadmin]

Hi, Thanks for getting back to me and for creating such a useful program. I am using the current version. Like I said when I upload the 20GB file I get that error. When I try to upload a 23gb file I get a "400 bad request"  however the 10GB file uploads fine. Both errors happen towards the end of the upload after it has an offset of near completion.Let me know if you have any questions. Hope you have a great vacation.Thanks

-------- Original Message --------
Subject: Re: [mt-aws-glacier] Large file upload (#4)
From: Victor Efimov notifications@github.com
Date: Sun, September 30, 2012 7:49 am
To: vsespb/mt-aws-glacier mt-aws-glacier@noreply.github.com
Cc: linuxadmin scott@theproblemfixer.comHello. I am currently on leave, so can be slow in responding. Are you sure yo're using latest version of the software ? (tag v0.7beta https://github.com/vsespb/mt-aws-glacier/zipball/v0.7beta == current master branch). Will try to reproduce with 20Gb when back from a vacation (next week). Thanks. — Reply to this email directly or view it on GitHub.

[vsespb]

Hello.

I was able to reproduce it.

(below is stack trace for latest v0.7beta version, just for clarity)

DIE outside EVAL block [0]
Call stack:
main::process_forks(./mtglacier.pl:103)
ChildWorker::process(./mtglacier.pl:195)
GlacierRequest::finish_multipart_upload(ChildWorker.pm:74)
main::ANON(GlacierRequest.pm:304)
Fatal Error: 0 Can't call method "header" on an undefined value at GlacierRequest.pm line 304, line 1456.
DIE outside EVAL block [0]
Call stack:
main::process_forks(./mtglacier.pl:103)
ParentWorker::process_task(./mtglacier.pl:233)
ParentWorker::wait_worker(ParentWorker.pm:49)
main::ANON(ParentWorker.pm:73)
Fatal Error: 0 Unexpeced EOF in Pipe at ParentWorker.pm line 73, line 1455.

I enabled verbose HTTP logging in code and it looks to me that it's Amazon problem (at least there are HTTP 500 in responses, which means problem on their side), so I posted a question on Amazon forums https://forums.aws.amazon.com/thread.jspa?threadID=106284&tstart=0

Before they answer, I 'll try to find a workaround (like maybe use less concurrent workers or do more retries of failed requests)

[vsespb]

Amazon is investigating this issue, but seems I found that with 20Gb file, number of parts to upload is ~ 9500 (while max limit is 10000) (I use 2 Mb part size). So I added option "--partsize" - you can specify 1,2,4,8,16 .. any power-of-two number.

Try number higher that 2. I was able to upload that file using --partsize=4 (internally script received four HTTP 500 before 5th success request). So --partsize=8 or 16 might work better.

[vsespb]

Amazon answered:

https://forums.aws.amazon.com/thread.jspa?messageID=390950

This can occur where you are attempting to complete an upload with many parts. If you receive a 500 from a complete >upload request in the future, we recommend that you call it again once or twice per minute until it succeeds. This call is >idempotent, so this will only ever result in a single Archive being stored. If you are running into this issue reliably, another >workaround is to try using a larger part size so you have fewer parts.

so I will change my code to perform (almost) unlimited number of retries when finishing multipart upload (currently there are 5 with delay 1 second)

[linuxadmin]

Perfect. I will test further when you do your next release with the number of retries update. Thanks again for patching this so quickly, and keeping me up to date.

[vsespb]

Done. 100 retries with progressive delay. I would still suggest you to use large "--partsize" (Amazon charges for each HTTP request so bigger part size - less charges).

[tpdinh]

I'm trying the newer version (0.74beta) to download 2 files and get this error. The older version is working fine for the same command line.

MT-AWS-Glacier, part of MT-AWS suite, Copyright (c) 2012 Victor Efimov http://mt-aws.com/ Version 0.74beta
PID 30845 Started worker
PID 30846 Started worker
PID 30847 Started worker
PID 30848 Started worker
Error:
POST http://glacier.us-east-1.amazonaws.com/-/vaults/TA/jobs
Authorization: AWS4-HMAC-SHA256 Credential=AKIAJBRI2XAQQ3RB3BFA/20121116/us-east-1/glacier/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-glacier-version, Signature=a361e4836c5408f87935cc805234ede97ef41ac56182888b5bff0e676752f2ac
Host: glacier.us-east-1.amazonaws.com
User-Agent: mt-aws-glacier/0.74beta (http://mt-aws.com/) libwww-perl/5.833
Content-Length: 191
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20121116T201753Z
X-Amz-Glacier-Version: 2012-06-01

{
"Type": "archive-retrieval",
"ArchiveId": "1mAFFVTVymMS50YLEhbfKh-czVlg8-rLeKY-H_7bLOL2nh9-Wo9Xv0zvQgt7I40XlBZmCaVo-mkX_Nv-WBHzrur5vzdbZPJwILjDAG9RlA3W3is3R_peimaW3JHuRrUjVCc1_btrug"
}\n
HTTP/1.1 403 Forbidden
Date: Fri, 16 Nov 2012 20:17:53 GMT
Content-Length: 800
Content-Type: application/json
Client-Date: Fri, 16 Nov 2012 20:17:53 GMT
Client-Peer: 72.21.195.182:80
Client-Response-Num: 1
X-Amzn-RequestId: OPJaeYQ8vIKudljdzghJueux53F0WpRKF1YlEnXG-qSIl2Q

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/-/vaults/TA/jobs\n\ncontent-type:application/x-www-form-urlencoded; charset=utf-8\nhost:glacier.us-east-1.amazonaws.com\nx-amz-date:20121116T201753Z\nx-amz-glacier-version:2012-06-01\n\ncontent-type;host;x-amz-date;x-amz-glacier-version\n361acdfa41b281d2...
(+ 288 more bytes not shown)

PARENT Exit

[vsespb]

fixed, issue #9

[mr--white]

I know this is an old thread but I hope it's ok to bring this up here. I have some files that are in the 2 - 3 TB range that I'm looking to upload. If this is supported, what partsize and concurrency would you recommend? Let's say the system has 96GB of RAM and 24 cores to work with.

[vsespb]

it should work with mtglacier, but I did not test with such a huge files.

limitations of amazon:

• maximum 10 000 parts
• uploaded part expires after 24h
• HTTP 408/500 on high concurrency (sometimes).
• HTTP 408/500 when uploading huge part (sometimes)

thus you need to:

• set partsize, I think 256 will work for 2TB (256MB * 10_000 = 2.4TB), but for 3TB you'll need partsize=512 (512Mb*10_000 = 4.8Tb).
  (mtglacier will warn you if you partsize is too before uploading file, no worries here)
• calculate you bandwidth, make sure you will able to upload whole file during 24h (Amazon tells: The ID is valid for at least 24 hours.)
• try upload this file with some --concurrency, like 3 or 4.
  make sure you are not getting (too much) HTTP errors and timeout on screen. make sure parts suceessfully uploaded (you see messages that is' uploaded)

if you're getting errors, you should try decrease concurrency.
if no errors, you can increase it to use bandwith more efficient (however - higher concurrency - more risk)

If mtglacier crash during upload (say, because of new, unknown bug), you'll have to begin from scratch. it does not remember uploaded parts in persistent storage.