Skip to content
This repository has been archived by the owner on Oct 30, 2018. It is now read-only.

Check Available RAM Before Setting Shard Size #411

Closed
super3 opened this issue Sep 9, 2016 · 1 comment
Closed

Check Available RAM Before Setting Shard Size #411

super3 opened this issue Sep 9, 2016 · 1 comment
Assignees

Comments

@super3
Copy link
Contributor

super3 commented Sep 9, 2016

Package Versions

Replace the values below using the output from npm list storj. Use npm list -g storj if installed globally.

├── storj@3.1.0
└─┬ storj-bridge-client@0.4.3
  └── storj@0.6.16

Replace the values below using the output from node --version.

v4.4.5

Expected Behavior

Upload proceeds until completion.

Actual Behavior

Shard size exceeded available memory.

 [Wed Aug 31 2016 13:44:57 GMT-0400 (Eastern Daylight Time)] [info]   [ dump2 ] Storing file, hang tight!
 [Wed Aug 31 2016 13:44:57 GMT-0400 (Eastern Daylight Time)] [debug]  Request: {"baseUrl":"https://api.staging.storj.io","uri":"/frames","method":"POST","json":{"__nonce":"93b2e57b-6ea0-4293-98e7-935e36dd5864"},"headers":{"x-pubkey":"03026d072b0612c7b419c16bfd811760e34551e9422a1c5a95d2516896b1fc124f","x-signature":"304502207a7ac5ac33183764b1ed96ad2e060b022c6399b0fb0496046d50299628f82611022100ed9d3ce15431e0c4a501cd7d64d4557a8c1af10741789c9d228a09e312a96f16"}}
 [Wed Aug 31 2016 13:44:57 GMT-0400 (Eastern Daylight Time)] [debug]  Body: {"user":"cats@storj.io","shards":[],"size":0,"locked":false,"created":"2016-08-31T17:44:46.990Z","id":"57c7178e274ee8b3253b69e2"}
 [Wed Aug 31 2016 13:44:57 GMT-0400 (Eastern Daylight Time)] [info]   Trying to upload shard C:\Users\work\AppData\Local\Temp\c6a8e567c74f index 0
 [Wed Aug 31 2016 13:45:12 GMT-0400 (Eastern Daylight Time)] [info]   Hash for this shard is: 20aa0cf37b5d71653333eb4a718f60f1d4f38262
 [Wed Aug 31 2016 13:45:51 GMT-0400 (Eastern Daylight Time)] [info]   Audit generation for shard done.
 [Wed Aug 31 2016 13:45:51 GMT-0400 (Eastern Daylight Time)] [info]   Waiting on a storage offer from the network...
 [Wed Aug 31 2016 13:45:51 GMT-0400 (Eastern Daylight Time)] [info]   Querying bridge for contract for 20aa0cf37b5d71653333eb4a718f60f1d4f38262 (retry: 0)
 [Wed Aug 31 2016 13:45:51 GMT-0400 (Eastern Daylight Time)] [debug]  Request: {"baseUrl":"https://api.staging.storj.io","uri":"/frames/57c7178e274ee8b3253b69e2","method":"PUT","json":{"hash":"20aa0cf37b5d71653333eb4a718f60f1d4f38262","size":2147483648,"index":0,"challenges":["ddcfce75440de584e22f45ba52b62f04c76e8ace1378881a8a55dd44ed5a0242","8bf9f20a9a63d2fca8e5149ea56e76a11d1ebcb4207f6a770a13293596a08e5f","e207a7b5048c1dbdcaaa97a09dad130e648a6a997c67d9220cb2abcfddf62b77"],"tree":["e6983a5f9d0936650f37e23e57929bb30e84a6c4","beacf5c7e9173c7c72a397ddb5f96946858fdc38","2212628d64dd521af144932157bf5022b495d6cc","b472a266d0bd89c13706a4132ccfb16f7c3b9fcb"],"exclude":[],"__nonce":"3c770145-5f8b-4a02-98a1-6c339cdc35e2"},"headers":{"x-pubkey":"03026d072b0612c7b419c16bfd811760e34551e9422a1c5a95d2516896b1fc124f","x-signature":"304502206b4dbb60d53756a985cc256e5d362df48db78558b05ce9cf4be4d80e73f0762b022100958896121bfa61f8f13c99fc2bc14224cdb7498b870e2819982760c783831e21"}}
 [Wed Aug 31 2016 13:45:52 GMT-0400 (Eastern Daylight Time)] [debug]  Body: {"hash":"20aa0cf37b5d71653333eb4a718f60f1d4f38262","token":"984b8f5cad136a1091adb4717dafd7e8d014558f","operation":"PUSH","farmer":{"protocol":"0.8.0-STAGING","address":"104.196.127.153","port":4000,"nodeID":"061bc46c36e33747e622639ecddfb8422a76746b","lastSeen":1472665541408}}
 [Wed Aug 31 2016 13:45:52 GMT-0400 (Eastern Daylight Time)] [info]   Contract negotiated with: {"protocol":"0.8.0-STAGING","address":"104.196.127.153","port":4000,"nodeID":"061bc46c36e33747e622639ecddfb8422a76746b","lastSeen":1472665541408}
 [Wed Aug 31 2016 13:45:52 GMT-0400 (Eastern Daylight Time)] [info]   Data channel opened, transferring shard...
 [Wed Aug 31 2016 13:48:15 GMT-0400 (Eastern Daylight Time)] [info]   Shard transfer completed! 49 remaining...
 [Wed Aug 31 2016 13:48:16 GMT-0400 (Eastern Daylight Time)] [info]   Trying to upload shard C:\Users\work\AppData\Local\Temp\2a8bad9d2e07 index 1
 [Wed Aug 31 2016 13:49:30 GMT-0400 (Eastern Daylight Time)] [info]   Hash for this shard is: 5378db53d7aeff8913801fb35d7f590fa395d949
 [Wed Aug 31 2016 13:51:08 GMT-0400 (Eastern Daylight Time)] [info]   Audit generation for shard done.
 [Wed Aug 31 2016 13:51:08 GMT-0400 (Eastern Daylight Time)] [info]   Waiting on a storage offer from the network...
 [Wed Aug 31 2016 13:51:08 GMT-0400 (Eastern Daylight Time)] [info]   Querying bridge for contract for 5378db53d7aeff8913801fb35d7f590fa395d949 (retry: 0)
 [Wed Aug 31 2016 13:51:08 GMT-0400 (Eastern Daylight Time)] [debug]  Request: {"baseUrl":"https://api.staging.storj.io","uri":"/frames/57c7178e274ee8b3253b69e2","method":"PUT","json":{"hash":"5378db53d7aeff8913801fb35d7f590fa395d949","size":2147483648,"index":1,"challenges":["1a2e42f6136816de43c1a7b572fe4ab1cce9346d0c3b5a2ca9e6a43f0744aa50","83a4d444b9f557c2504578c61eb5443c4b31c2e7543c3351922543e57e66a05d","a49e1230d5bf43e8fcb39238b74fab1cf91e27d22d2e49ca4a715b52eeb2f5c0"],"tree":["144bda546d289c7b95dbe3fc28d1ea25c40eb290","102554b9be43252c52cb1bbc36e1597355721910","8fff483aaa1cda6f926d138d872e1240deb4eaae","b472a266d0bd89c13706a4132ccfb16f7c3b9fcb"],"exclude":[],"__nonce":"14e7f503-1fec-4384-be33-6080598768cf"},"headers":{"x-pubkey":"03026d072b0612c7b419c16bfd811760e34551e9422a1c5a95d2516896b1fc124f","x-signature":"3045022016aedbd755cca191601d39c3198a46c127e6673840f5bea17b91f507569e419c022100a4d5b7c2aaf5565b9e63e1902c5245d05a0d3d18d6e47afdf5e4c3742aaa0878"}}
 [Wed Aug 31 2016 13:51:09 GMT-0400 (Eastern Daylight Time)] [debug]  Body: {"hash":"5378db53d7aeff8913801fb35d7f590fa395d949","token":"a1320812f31a4a76cc6d73c42d1588751d1ec1b7","operation":"PUSH","farmer":{"protocol":"0.8.0-STAGING","address":"104.196.117.75","port":4000,"nodeID":"6412ecb5763de95452ebf3273b4fa43f746efb74","lastSeen":1472665858389}}
 [Wed Aug 31 2016 13:51:09 GMT-0400 (Eastern Daylight Time)] [info]   Contract negotiated with: {"protocol":"0.8.0-STAGING","address":"104.196.117.75","port":4000,"nodeID":"6412ecb5763de95452ebf3273b4fa43f746efb74","lastSeen":1472665858389}
 [Wed Aug 31 2016 13:51:09 GMT-0400 (Eastern Daylight Time)] [info]   Data channel opened, transferring shard...
buffer.js:98
    const ui8 = new Uint8Array(size);
                ^

RangeError: Invalid array buffer length
    at new ArrayBuffer (native)
    at new Uint8Array (native)
    at allocate (buffer.js:98:17)
    at new Buffer (buffer.js:49:12)
    at allocNewPool (fs.js:1685:10)
    at ReadStream._read (fs.js:1784:5)
    at ReadStream.Readable.read (_stream_readable.js:328:10)
    at readableAddChunk (_stream_readable.js:147:16)
    at ReadStream.Readable.push (_stream_readable.js:110:10)
    at onread (fs.js:1822:12)
    at FSReqWrap.wrapper [as oncomplete] (fs.js:614:17)

Steps to Reproduce

Please include the steps the reproduce the issue, numbered below. Include as
much detail as possible.

  1. Upload 100 GB file for the lulz
  2. Crashes after 1-2 shards
@MeijeSibbel
Copy link

This error persists even with core 3.2.0, although i only get this towards the end of a upload process, or after say 3-4 large uploads while uploading a folder content.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants