Large files cause jsDAV to hard shutdown #83

Open
briangonzalez opened this Issue Aug 13, 2013 · 9 comments

Projects

None yet

3 participants

@briangonzalez

I have been uploading large files (>2GB) and the files seem to complete, but at the very end of the transfer, I get a 502 Bad Gateway in my webDAV client.

Looking at std out shows that jsDAV has in fact killed itself.

Here's the output right before jsDAV outputs "Killed"

[info] {358} PROPFIND /
[info] {358} { host: 'localhost:9000',
[info]   connection: 'close',
[info]   'content-length': '179',
[info]   'content-type': 'text/xml',
[info]   depth: '0',
[info]   accept: '*/*',
[info]   'user-agent': 'WebDAVFS/3.0.0 (03008000) Darwin/13.0.0 (x86_64)' }
[info] Receiving headers: {"host":"localhost:9000","connection":"close","content-length":"179","content-type":"text/xml","depth":"0","accept":"*/*","user-agent":"WebDAVFS/3.0.0 (03008000) Darwin/13.0.0 (x86_64)"}
[info] Returning headers: {"Access-Control-Allow-Methods":"GET, HEAD, PUT, POST, DELETE, TRACE, OPTIONS, CONNECT, PATCH, PROPFIND, PROPPATCH, MKCOL, COPY, MOVE, LOCK, UNLOCK, VERSION-CONTROL, REPORT, CHECKOUT, CHECKIN, UNCHECKOUT, MKWORKSPACE, UPDATE, LABEL, MERGE, BASELINE-CONTROL, MKACTIVITY, GETLIB","Access-Control-Max-Age":"86400","Access-Control-Allow-Headers":["accept","accept-charset","accept-encoding","accept-language","authorization","content-length","content-type","host","origin","proxy-connection","referer","user-agent","x-requested-with"],"Access-Control-Allow-Credentials":"true","Access-Control-Allow-Origin":"*"}
[info] {358} <?xml version="1.0" encoding="utf-8"?>
[info] <D:propfind xmlns:D="DAV:">
[info] <D:prop>
[info] <D:getlastmodified/>
[info] <D:getcontentlength/>
[info] <D:creationdate/>
[info] <D:resourcetype/>
[info] </D:prop>
[info] </D:propfind>
[info]
[info] {358} 207 { 'content-type': 'application/xml; charset=utf-8',
[info]   vary: 'Brief,Prefer',
[info]   DAV: '1,3,extended-mkcol,2' }
[info] {358} '<?xml version="1.0" encoding="utf-8"?><d:multistatus xmlns:d="DAV:" xmlns:a="http://ajax.org/2005/aml"><d:response><d:href>/</d:href><d:propstat><d:prop><d:getlastmodified xmlns:b="urn:uuid:c2f41010-65b3-11d1-a29f-00aa00c14882/" b:dt="dateTime.rfc1123">Mon, 12 Aug 2013 17:37:10 -0700</d:getlastmodified><d:getcontentlength></d:getcontentlength><d:creationdate></d:creationdate><d:resourcetype><d:collection/></d:resourcetype></d:prop><d:status>HTTP/1.1 200 Ok</d:status></d:propstat></d:response></d:multistatus>'
Killed
@ooxi

Is this reproduceable? Does it depend on the file contents? Is 2 GB the exact border when this starts happening? How much RAM do you have?

@briangonzalez

Yes, reproducible in that it happens every upload. of a "large" file. I am not sure about the file's contents, but I have mostly been testing with .zip files.

I has jsDAV running on a an RPi with 512MB of memory.

I am going to test files of the following sizes generated using the shell command: dd if=/dev/zero of=output_400MB.dat bs=1024 count=400240. Not that these are all .dat files.

Files

  • 400MB (just under memory cap) - SUCCESS
  • 600MB (just over memory cap.) - SUCCESS
  • 750MB - SUCCESS
  • 850MB - FAIL
  • 1GB - FAIL
  • 1.5GB - FAIL
@briangonzalez

Tailed the syslog, and look what I've found:

# tail -f /var/log/syslog --lines 5
Aug 13 07:53:18 raspberrypi kernel: [121175.599784] [16287]     0 16287     1219        8       6       98             0 bash
Aug 13 07:53:18 raspberrypi kernel: [121175.599801] [16344]     0 16344     1121       19       6       44             0 screen
Aug 13 07:53:18 raspberrypi kernel: [121175.599819] [16374]     0 16374   137706   105056     265    18933             0 nodejs
Aug 13 07:53:18 raspberrypi kernel: [121175.599833] Out of memory: Kill process 16374 (nodejs) score 871 or sacrifice child
Aug 13 07:53:18 raspberrypi kernel: [121175.599849] Killed process 16374 (nodejs) total-vm:550824kB, anon-rss:419436kB, file-rss:788kB
@briangonzalez

@ooxi Any idea what could be going on here?

@mikedeboer
Owner

@briangonzalez how are you uploading files? Through a browser POST form or using a file manager with PUT?

@mikedeboer
Owner

Oh, and also an important question: which version of jsDAV are you using?

@briangonzalez

I am using a clone of master at 558377a.

Also, I am using Transmit and Cyberduck on the Mac, both of which have the same issue.

@mikedeboer mikedeboer was assigned Oct 2, 2013
@mikedeboer
Owner

Ugh, I have so little time :( I will take a look at this, because it's something a NodeJS lib should do well!

@briangonzalez

No worries, take your time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment