Allocation failed

[Lunik]

Hi,
I'm trying to transfert large file with webtorrent with client.add function.
But it crash before the callback was called, with some allocation issues

What version of WebTorrent?
0.98.19

What operating system and Node.js version?
Centos 7, node v8.5.0

What did you expect to happen?
Download to start

What actually happened?
Crash with allocation failed

Trace

(node:27) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 ready listeners added. Use emitter.setMaxListeners() to increase limit
FATAL ERROR: SequentialMarkingDeque::EnsureCommitted Allocation failed - process out of memory

<--- Last few GCs --->

[27:0x56195c757760]   221963 ms: Mark-sweep 164.6 (202.0) -> 157.3 (202.0) MB, 53.1 / 4.4 ms  (+ 704.3 ms in 676 steps since start of marking, biggest step 5.2 ms, walltime since start of marking 1011 ms) finalize incremental marking via task GC in old sp[27:0x56195c757760]   222183 ms: Scavenge 171.9 (202.0) -> 157.2 (202.0) MB, 15.3 / 0.3 ms  allocation failure 
[27:0x56195c757760]   223103 ms: Scavenge 172.4 (202.0) -> 158.1 (202.0) MB, 19.2 / 0.9 ms  allocation failure 


<--- JS stacktrace --->
Cannot get stack trace in GC.

<--- Last few GCs --->

[1:0x55e64b817e40]   270743 ms: Scavenge 25.5 (31.6) -> 24.7 (31.6) MB, 2.8 / 0.1 ms  allocation failure 
[1:0x55e64b817e40]   298818 ms: Scavenge 25.6 (31.6) -> 24.9 (31.6) MB, 4.3 / 0.0 ms  allocation failure 
[1:0x55e64b817e40]   324741 ms: Scavenge 25.7 (31.6) -> 25.0 (31.6) MB, 3.0 / 0.0 ms  allocation failure 
[1:0x55e64b817e40]   344738 ms: Scavenge 25.9 (31.6) -> 25.2 (31.6) MB, 3.6 / 0.0 ms  allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1c676dea8799 <JSObject>
    2: InnerArraySort(aka InnerArraySort) [native array.js:1] [bytecode=0x1f94d220a6f9 offset=12](this=0x1165af382311 <undefined>,p=0x2f4a9ad7ff61 <JSArray[1]>,q=1,aB=0x1165af382311 <undefined>)
    4: sort [native array.js:1] [bytecode=0x1f94d2209a41 offset=57](this=0x2f4a9ad7ff61 <JSArray[1]>,aB=0x1165af382311 <undefined>)
    5: arguments adaptor frame: 0->1
    7: format [/usr/src/app/node_m...

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory

[Lunik]

Is using --max_old_space_size option relevant ?

[disarticulate]

I've been documenting in the browser, memory usage information here:

#895 (comment)

The only solution I came to reduce memory usage that's leaking in the browser is to add pieceLength to the options. for 31.6MB something like 1-2 mb (1000000-2000000) should fix it if it's related to the memory leak.

[feross]

@Lunik How many torrents are you adding? What is the size of each torrent?

[no-response]

This issue has been automatically closed because there was no response to the maintainer's request for more information from the issue opener. With only the information that is currently in the issue, we don't have enough information to take action. Please leave a comment or open a new issue if you have the additional information we need to investigate further.

[ziro-official]

Hi @feross

I tried to download a torrent, 4GB file size on a VPS, 1GB Ram (for testing purpose). Got this issue:

CPU model            : Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz
Number of cores      : 2
CPU frequency        : 1066.738 MHz
Total size of Disk   : 20.0 GB (13.0 GB Used)
Total amount of Mem  : 1024 MB (445896 Bytes Free)
Total amount of Swap : 0 MB (0 MB Used)
System uptime        : 0 days, 3 hour 21 min
Load average         : 0.13, 0.15, 0.05
OS                   : Ubuntu 16.04 LTS
Arch                 : x86_64 (64 Bit)
Kernel               : 2.6.32-042stab123.1

<--- Last few GCs --->

[13330:0x2f62a30]   143288 ms: Scavenge 18.3 (24.4) -> 16.4 (24.4) MB, 0.6 / 0.0 ms  allocation failure 
[13330:0x2f62a30]   143345 ms: Mark-sweep 16.9 (24.4) -> 16.0 (24.4) MB, 33.2 / 0.0 ms  allocation failure GC in old space requested
[13330:0x2f62a30]   143374 ms: Mark-sweep 16.0 (24.4) -> 15.9 (24.4) MB, 29.6 / 0.0 ms  allocation failure GC in old space requested
[13330:0x2f62a30]   143407 ms: Mark-sweep 15.9 (24.4) -> 15.9 (21.4) MB, 32.9 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2da4d5b25ee1 <JSObject>
    1: onread [net.js:~576] [pc=0x2a281a058021](this=0x19c116efe799 <TCP map = 0x3d9e4220b4c9>,nread=1460,buffer=0x41c6e2edeb1 <Uint8Array map = 0x3d9e42208501>)
    2: arguments adaptor frame: 3->2

==== Details ================================================

[1]: onread [net.js:~576] [pc=0x2a281a058021](this=0x19c116efe799 <TCP map = 0x3d9e4220b4c9>,nread=1460,buffer=0x41c6e2edeb1 <Uint8Arra...

FATAL ERROR: Committing semi space failed. Allocation failed - process out of memory
 1: node::Abort() [node]
 2: 0x121a2cc [node]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 6: 0xea7f33 [node]
 7: v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [node]
 8: v8::internal::Factory::NewByteArray(int, v8::internal::PretenureFlag) [node]
 9: v8::internal::TranslationBuffer::CreateByteArray(v8::internal::Factory*) [node]
10: v8::internal::compiler::CodeGenerator::PopulateDeoptimizationData(v8::internal::Handle<v8::internal::Code>) [node]
11: v8::internal::compiler::CodeGenerator::FinalizeCode() [node]
12: v8::internal::compiler::PipelineImpl::FinalizeCode() [node]
13: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [node]
14: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [node]
15: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
16: v8::internal::StackGuard::HandleInterrupts() [node]
17: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
18: 0x2a2819e0463d
Aborted