Memory leak? WebTorrent API eats up 6gb RAM, then crashes with "Out of memory"

[tsoernes]

Hi.
(Webtorrent) v0.62.3, node v5.0.0)
I'm running the following code:

       client = new WebTorrent();
    client.add(torrentFilePath, function ontorrent (torrent) {
        var server = torrent.createServer();

        server.listen(8888);
        biggestIdx = 0;
        for (var i=1; i<torrent.files.length; i++) {
            if (torrent.files[i].length > torrent.files[biggestIdx].length) {
                biggestIdx = i;
            }
        }
        var url = "http://localhost:8888/" + biggestIdx
        console.log(torrent.files[biggestIdx].name + " available at " + url);
    });

When the code is called, the V8 engine eats up more than 6 GB RAM (in a couple of seconds) and then crashes: FATAL ERROR: node::smalloc::Alloc(v8::Handle<v8::Object>, size_t, v8::ExternalArrayType) Out Of Memory
After upgrading from Node v0.14 to v5.0.0, the error is changed to this:

buffer.js:98
    const ui8 = new Uint8Array(size);
                ^

RangeError: Invalid array buffer length
    at new ArrayBuffer (native)
    at new Uint8Array (native)
    at allocate (buffer.js:98:17)
    at new Buffer (buffer.js:49:12)
    at alloc (\node_modules\random-access-file\index.js:14:32)
    at node_modules\random-access-file\index.js:77:20
    at apply (node_modules\thunky\index.js:16:28)
    at \node_modules\thunky\index.js:20:25
    at onfinish (\leechflix\node_modules\random-access-file\index.js:37:4)
    at \node_modules\random-access-file\index.js:43:49

The torrent has 8GB of files.
I tried with a smaller torrent. It runs without error, but uses a lot of memory. WebTorrent is the only thing running in the code:

[blairanderson]

I think because of browser capabilities, the library loads the entire file into memory before seeding. since your ram is less than the file size it will not work.

[yas9999]

@blairanderson 👍 does webtorrent control file-size in order to limit users to download allowed size !

[tsoernes]

@blairanderson is there a way to avoid that? Torrentstream (peerflix) does
not use that much memory
On Nov 22, 2015 18:46, "yas9999" notifications@github.com wrote:

@blairanderson https://github.com/blairanderson [image: 👍] does
webtorrent control file-size in order to limit users to download allowed
size !

—
Reply to this email directly or view it on GitHub
#479 (comment).

[yas9999]

i guess u can use https://github.com/feross/webtorrent-hybrid.

[tsoernes]

I thought the only difference with hybrid was webrtc support? How would
that help?
On Nov 26, 2015 19:10, "yas9999" notifications@github.com wrote:

i guess u can use https://github.com/feross/webtorrent-hybrid.

—
Reply to this email directly or view it on GitHub
#479 (comment).

[rom1504]

maybe look at #248
It seems webtorrent in node shouldn't be using that much memory.

(and no webtorrent-hybrid wouldn't help for this issue)

[DiegoRBaquero]

@tsoernes Any changes since the implementation of peer destorying?

[tsoernes]

No changes:

DEBUG INFO: webtorrent 0.63.4, node v5.0.0, win32 x64, exit 1
buffer.js:98
    const ui8 = new Uint8Array(size);
                ^
RangeError: Invalid array buffer length

I don't know how peer destroying, whatever that is, would help. It crashes when loading the torrent, before finding peers.

[feross]

@tsoernes I'm taking a look at this issue now.

Is there any chance you could re-run with the latest version of webtorrent (0.72.1) and enable debug logs? You can do that by setting the DEBUG environment variable to *. In unix, you do that like this:

DEBUG=* node myscript.js

Thanks!

[feross]

I can't reproduce the issue with the latest version of webtorrent or the version you were using. I'm using your sample code and a torrent with really large files (25GB).

@tsoernes Your debug logs would really help here.

[feross]

We can't reproduce this issue without debug logs. Closing.

[ali-khabbaz]

i tried to download 4 gb file, i got this error :

`Started saving English(SDH).srt
Started saving English.srt
Started saving French.srt
Started saving Spanish.srt
Started saving Star.Wars.Episode.VII.The.Force.Awakens.2015.1080p.BluRay.x264.DTS-JYK.mkv
buffer.js:23
const ui8 = new Uint8Array(size);
^

RangeError: Invalid typed array length
at new Uint8Array (native)
at createBuffer (buffer.js:23:15)
at allocate (buffer.js:98:12)
at new Buffer (buffer.js:53:12)
at getBuffer (/root/node_modules/webtorrent/node_modules/stream-with-known-length-to-buffer/index.js:5:13)
at File.getBuffer (/root/node_modules/webtorrent/lib/file.js:96:3)
at /root/test/webtorrent-hybrid.js:15:8
at Array.forEach (native)
at /root/test/webtorrent-hybrid.js:12:16
at Torrent.onReady (/root/node_modules/webtorrent/index.js:215:42)
`

debug logs:

bittorrent-dht [92cedd5] new DHT 92cedd51f8056e0203528496d08b644e46b7f525 +0ms webtorrent new webtorrent (peerId 2d5757303039302d383266636563643861633937, nodeId 92cedd51f8056e0203528496d08b644e46b7f525) +6ms webtorrent add +3ms webtorrent:torrent [61633937] new torrent +4ms bittorrent-dht [92cedd5] listening 56495 +10ms webtorrent:swarm new swarm (i 361c3b67f562f2e7f4d10ea22022d26beda88286 p 2d5757303039302d383266636563643861633937) +7ms webtorrent:swarm listen 0 +1ms webtorrent:tcp-pool new TCPPool (port: 0, hostname: undefined) +0ms webtorrent:tcp-pool add swarm 361c3b67f562f2e7f4d10ea22022d26beda88286 to tcp pool 0 +3ms webtorrent:tcp-pool tcp pool listening on 40573 +1ms bittorrent-tracker new client 361c3b67f562f2e7f4d10ea22022d26beda88286 +3ms bittorrent-tracker:udp-tracker new udp tracker udp://glotorrents.pw:6969/announce +4ms bittorrent-tracker:udp-tracker new udp tracker udp://tracker.openbittorrent.com:80/announce +1ms bittorrent-tracker:udp-tracker new udp tracker udp://tracker.publicbt.com:80/announce +0ms bittorrent-tracker:websocket-tracker new websocket tracker wss://tracker.btorrent.xyz +1ms simple-websocket new websocket: wss://tracker.btorrent.xyz {} +0ms bittorrent-tracker:websocket-tracker new websocket tracker wss://tracker.fastcast.nz +61ms simple-websocket new websocket: wss://tracker.fastcast.nz {} +0ms bittorrent-tracker:websocket-tracker new websocket tracker wss://tracker.openwebtorrent.com +11ms simple-websocket new websocket: wss://tracker.openwebtorrent.com {} +0ms bittorrent-tracker:websocket-tracker new websocket tracker wss://tracker.webtorrent.io +2ms simple-websocket new websocket: wss://tracker.webtorrent.io {} +1ms bittorrent-tracker setInterval 900000 +2ms bittorrent-tracker sendstart` +1ms
torrent-discovery dht announce +7ms
bittorrent-dht [92cedd5] lookup 361c3b67f562f2e7f4d10ea22022d26beda88286 +1ms
events.js:141
throw er; // Unhandled 'error' event
^

Error: write EPIPE
at exports._errnoException (util.js:870:11)
at WriteWrap.afterWrite (net.js:769:14)
`

[feross]

@ali-khabbaz Do not post links or reference to copyrighted material that you do not own. Do not use WebTorrent for copyright infringement.

[lock]

This thread has been automatically locked because it has not had recent activity. To discuss futher, please open a new issue.