Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bcoin full validation sync crash - heap out of memory #626

Closed
jlopp opened this issue Nov 1, 2018 · 21 comments

Comments

@jlopp
Copy link
Contributor

commented Nov 1, 2018

I tried running a full validation sync on PureOS 8.0 (Debian variant) with the following config:

bcoin --max-files 640 --cache-size 640 --checkpoints false --coin-cache 4000

Perhaps one of my configs caused the heap to be exceeded? If so, you may need some sanity checks as ceilings on these configs. Here's the stack trace:

[debug] (chain) Coin Cache: size=387.69784450531006mb, items=1764670.
[debug] (net) Status: time=2012-07-02T13:32:58Z height=187200 progress=35.55% orphans=10 active=300 target=436844426 peers=8
[debug] (chain) Memory: rss=4089mb, js-heap=945/1405mb native-heap=2684mb
[info] (chain) Block 000000000000084c95101eaa9ba7261a5fc8eeee081ada062e328e7faba7ccfb (187220) added to chain (size=163123 txs=400 time=63.951539).
[debug] (chain) Coin Cache: size=388.04532527923584mb, items=1766262.
[debug] (net) Status: time=2012-07-02T15:53:50Z height=187220 progress=35.55% orphans=10 active=280 target=436844426 peers=8
[debug] (peer) Peer has not responded to ping (193.106.30.173:8333).
[debug] (chain) Memory: rss=4104mb, js-heap=959/1411mb native-heap=2693mb
[info] (chain) Block 0000000000000644f685abde12293857e7497499c6d7e3bb532bfa9b1b272862 (187240) added to chain (size=119456 txs=277 time=39.051175).
[debug] (chain) Coin Cache: size=388.0974750518799mb, items=1766512.
[debug] (net) Status: time=2012-07-02T20:03:44Z height=187240 progress=35.56% orphans=10 active=260 target=436844426 peers=8
[debug] (chain) Memory: rss=4111mb, js-heap=960/1414mb native-heap=2697mb
[info] (chain) Block 0000000000000247aae495abc2d893dc00c068949f63530dafef8e93f2afd1d5 (187260) added to chain (size=249069 txs=704 time=136.125399).
[debug] (chain) Coin Cache: size=388.3010025024414mb, items=1767448.
[debug] (net) Status: time=2012-07-03T00:10:56Z height=187260 progress=35.56% orphans=10 active=240 target=436844426 peers=8
[debug] (chain) Memory: rss=4111mb, js-heap=919/1414mb native-heap=2697mb
[info] (chain) Block 000000000000074b9eb41b468b0036b6c81e0d36b85f2523a429ec52a41412d6 (187280) added to chain (size=65001 txs=99 time=54.022798).
[debug] (chain) Coin Cache: size=388.32544708251953mb, items=1767566.
[debug] (net) Status: time=2012-07-03T04:53:04Z height=187280 progress=35.57% orphans=10 active=220 target=436844426 peers=8

<--- Last few GCs --->

[17190:0x5575463b39b0]  5181949 ms: Mark-sweep 904.8 (1401.0) -> 904.5 (1400.5) MB, 743.6 / 0.0 ms  allocation failure GC in old space requested
[17190:0x5575463b39b0]  5182720 ms: Mark-sweep 904.5 (1400.5) -> 904.5 (1342.5) MB, 771.5 / 0.0 ms  last resort GC in old space requested
[17190:0x5575463b39b0]  5183486 ms: Mark-sweep 904.5 (1342.5) -> 904.5 (1318.5) MB, 765.6 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x7302d599099 <JSObject>
    1: set(this=0xdfe5b8597d9 <Map map = 0x99bbfe93a29>,0x825c9f3b041 <String[36]\: o\xb1@b\x00\x8d\xc6\x93Ah\x87a\\@\x8d\xaa^\xe2\xa09rC1\x8d\x02Pki\x1a\x92FG\x01\x00\x00\x00>,0x825c9f3b081 <BufferItem map = 0x1f1bc706be49>)
    2: set [/home/jameson/Downloads/bcoin/node_modules/blru/lib/lru.js:~119] [pc=0x25214eed056c](this=0x32660f517dd1 <LRU map = 0x1f1bc7042db9>,key=0x825c9f4f9e9 <Uint8Array...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bcoin]
 2: 0x557543baff91 [bcoin]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [bcoin]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bcoin]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [bcoin]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [bcoin]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [bcoin]
 8: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [bcoin]
 9: 0x25214e6040bd
Aborted

I then restart bcoin with the same settings to see if it was reproducible and it died about 20 minutes later:

[debug] (chain) Coin Cache: size=441.9780435562134mb, items=2019753.
[debug] (net) Status: time=2013-01-19T10:50:19Z height=217140 progress=41.15% orphans=3 active=242 target=436545969 peers=8
[info] (net) Received 1 addrs (hosts=1000, peers=8) (185.21.216.134:8333).
[debug] (peer) Peer has not responded to ping (190.10.8.211:8333).
[debug] (chain) Memory: rss=4401mb, js-heap=938/1454mb native-heap=2947mb
[info] (chain) Block 00000000000001f363f281e54867b1190edfdab9239a5530c245d9bd59eeb4fd (217160) added to chain (size=262760 txs=447 time=914.415159).
[debug] (chain) Coin Cache: size=442.30596923828125mb, items=2021248.
[debug] (net) Status: time=2013-01-19T14:38:15Z height=217160 progress=41.15% orphans=3 active=222 target=436545969 peers=8
[info] (net) Received 1 addrs (hosts=1000, peers=8) (185.21.216.134:8333).
[debug] (chain) Memory: rss=4404mb, js-heap=931/1454mb native-heap=2949mb
[info] (chain) Block 000000000000008f64569d18d2fd2ca2cf33e459f2e63fcfa2a96a073b5526c5 (217180) added to chain (size=162921 txs=404 time=826.045948).
[debug] (chain) Coin Cache: size=442.41479873657227mb, items=2021739.
[debug] (net) Status: time=2013-01-19T17:10:00Z height=217180 progress=41.15% orphans=3 active=202 target=436545969 peers=8
[info] (net) Received 2 addrs (hosts=1000, peers=8) (190.10.8.211:8333).
[debug] (peer) Peer has not responded to ping (190.10.8.211:8333).
[info] (net) Received 1 addrs (hosts=1000, peers=8) (190.10.8.211:8333).
[debug] (chain) Memory: rss=4413mb, js-heap=939/1455mb native-heap=2957mb
[info] (chain) Block 00000000000003bd639654d6e78a4357b85c887c39c608c7e43abfb129aef6a7 (217200) added to chain (size=16092 txs=43 time=8.314394).
[debug] (chain) Coin Cache: size=442.70891761779785mb, items=2023083.
[debug] (net) Status: time=2013-01-19T20:00:15Z height=217200 progress=41.16% orphans=3 active=182 target=436545969 peers=8
[info] (net) Received 1 addrs (hosts=1000, peers=8) (169.44.34.203:8333).
[debug] (chain) Memory: rss=4420mb, js-heap=935/1456mb native-heap=2964mb
[info] (chain) Block 000000000000028ce07f94a6ac7cbb88a82104a4d1ed47cd95c8ee7bbfc83145 (217220) added to chain (size=231603 txs=638 time=1606.546598).
[debug] (chain) Coin Cache: size=443.2419710159302mb, items=2025518.
[debug] (net) Status: time=2013-01-19T22:49:55Z height=217220 progress=41.16% orphans=3 active=162 target=436545969 peers=8
[debug] (peer) Peer has not responded to ping (190.10.8.211:8333).

<--- Last few GCs --->

[17645:0x5593de6699b0]  2537391 ms: Mark-sweep 930.5 (1458.5) -> 930.3 (1458.5) MB, 1144.5 / 0.0 ms  allocation failure GC in old space requested
[17645:0x5593de6699b0]  2538109 ms: Mark-sweep 930.3 (1458.5) -> 930.3 (1403.5) MB, 718.0 / 0.0 ms  last resort GC in old space requested
[17645:0x5593de6699b0]  2538927 ms: Mark-sweep 930.3 (1403.5) -> 930.3 (1385.0) MB, 817.9 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x156d16219099 <JSObject>
    1: set(this=0x36cad9ac6b69 <Map map = 0x375053513a29>,0x327e6a15ab39 <String[36]\: %wy\xe4\x0e\x18{<\x1f\xf1o^P\xabr\xd2A\xb6\xba\x19\xd9^\x9b<\xc0\xe6\xd8\xd8d\xdbl\\\x00\x00\x00\x00>,0x327e6a15ab79 <BufferItem map = 0x2d80f94605b1>)
    2: set [/home/jameson/Downloads/bcoin/node_modules/blru/lib/lru.js:~119] [pc=0xad234090d0f](this=0x36cad9abe3a1 <LRU map = 0x2d80f9442db9>,key=0x2c55b7dc3b...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bcoin]
 2: 0x5593dcdc9f91 [bcoin]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [bcoin]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bcoin]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [bcoin]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [bcoin]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [bcoin]
 8: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [bcoin]
 9: 0xad2332840bd
Aborted

I then ran "bcoin --checkpoints false --coin-cache 4000" to check that this wasn't an issue with the max-files or cache-size flags and also reproduced the heap OOM error:

[debug] (chain) Coin Cache: size=539.1922397613525mb, items=2452950.
[debug] (net) Status: time=2013-04-30T11:56:36Z height=233900 progress=43.96% orphans=10 active=358 target=436316733 peers=8
[debug] (peer) Peer has not responded to ping (193.106.30.173:8333).
[info] (net) Received 1 addrs (hosts=1000, peers=8) (69.64.46.27:8333).
[info] (net) Received 2 addrs (hosts=1000, peers=8) (149.210.228.108:8333).
[debug] (chain) Memory: rss=3654mb, js-heap=1152/1574mb native-heap=2079mb
[info] (chain) Block 00000000000001485853fb76792a5fdc0df2809335c74960fefc214e034f4faf (233920) added to chain (size=305781 txs=593 time=2125.159939).
[debug] (chain) Coin Cache: size=542.7767372131348mb, items=2469425.
[debug] (net) Status: time=2013-04-30T15:01:27Z height=233920 progress=43.96% orphans=10 active=338 target=436316733 peers=8
[info] (net) Received 1 addrs (hosts=1000, peers=8) (185.21.216.134:8333).

<--- Last few GCs --->

[17900:0x560468e29910]  3418883 ms: Mark-sweep 1140.1 (1576.0) -> 1140.1 (1576.0) MB, 1218.4 / 0.0 ms  allocation failure GC in old space requested
[17900:0x560468e29910]  3419822 ms: Mark-sweep 1140.1 (1576.0) -> 1140.1 (1527.5) MB, 938.5 / 0.0 ms  last resort GC in old space requested
[17900:0x560468e29910]  3420779 ms: Mark-sweep 1140.1 (1527.5) -> 1140.1 (1522.5) MB, 956.5 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x22af9c499099 <JSObject>
    2: fromRaw [/home/jameson/Downloads/bcoin/lib/script/script.js:3272] [bytecode=0x291fad5e9e81 offset=0](this=0x39e08f9cc3e1 <Script map = 0x1fb27e3b2c99>,data=0x39e08f9cb769 <Uint8Array map = 0x35592875b019>)
    3: fromReader [/home/jameson/Downloads/bcoin/lib/primitives/input.js:401] [bytecode=0x291fad5e9861 offset=42](this=0x39e08f9cc381 <Input map = 0x1fb27e3b2df9>,br=0xe7a880b2331 <Buff...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bcoin]
 2: 0x56046756df91 [bcoin]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [bcoin]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bcoin]
 5: v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, unsigned int, v8::internal::Handle<v8::internal::Object>, bool, int) [bcoin]
 6: v8::internal::CodeGenerator::MakeCodeEpilogue(v8::internal::TurboAssembler*, v8::internal::EhFrameWriter*, v8::internal::CompilationInfo*, v8::internal::Handle<v8::internal::Object>) [bcoin]
 7: v8::internal::compiler::CodeGenerator::FinalizeCode() [bcoin]
 8: v8::internal::compiler::PipelineImpl::FinalizeCode() [bcoin]
 9: v8::internal::compiler::PipelineCompilationJob::FinalizeJobImpl() [bcoin]
10: v8::internal::CompilationJob::FinalizeJob() [bcoin]
11: v8::internal::Compiler::FinalizeCompilationJob(v8::internal::CompilationJob*) [bcoin]
12: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [bcoin]
13: v8::internal::StackGuard::HandleInterrupts() [bcoin]
14: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [bcoin]
15: 0x1c20510840bd
Aborted
@nodar-chkuaselidze

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2018

That limitation comes from v8, by default it's set to ~1.5GB.
You can modify that limitation using --max-old-space-size, e.g. 8192.

If you want to verify what's limit for your v8, you can use https://nodejs.org/api/v8.html#v8_v8_getheapspacestatistics

@jlopp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 1, 2018

It sounds like there should be a sanity check during initialization to ensure that the coin-cache size is less than the heap size, right?

@nodar-chkuaselidze

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2018

Well, you can do that, but that won't guarantee that you won't run out of memory, because there are many things that consume heap. (mempool, block process, net stack...).

But definitely can be improved with docs and maybe additional params and sanity checks.

@jlopp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 2, 2018

OK so I tried running: bcoin --max-files 640 --cache-size 640 --checkpoints false --coin-cache 10000 --max-old-space-size 16000

After 2 hours it ran out of memory:

[debug] (chain) Memory: rss=4570mb, js-heap=904/1412mb native-heap=3158mb
[info] (chain) Block 000000000000053ab5cafe0435882b6cae116e69be95c0b10968ea94d9ba2d55 (204200) added to chain (size=310327 txs=584 time=788.171831).
[debug] (chain) Coin Cache: size=426.7566967010498mb, items=1947183.
[debug] (net) Status: time=2012-10-20T21:13:11Z height=204200 progress=38.62% orphans=3 active=179 target=436565487 peers=8
[debug] (chain) Memory: rss=4576mb, js-heap=905/1410mb native-heap=3166mb
[info] (chain) Block 00000000000000bf40184fed602c46ef3229afd9cb13dc8bcfcc6babd52e0141 (204220) added to chain (size=32699 txs=113 time=20.852169).
[debug] (chain) Coin Cache: size=427.25597381591797mb, items=1949462.
[debug] (net) Status: time=2012-10-21T00:41:39Z height=204220 progress=38.62% orphans=3 active=159 target=436565487 peers=8

<--- Last few GCs --->

[3966:0x55ad76a59a00]  1372475 ms: Mark-sweep 896.3 (1414.0) -> 895.9 (1411.5) MB, 793.2 / 0.0 ms  allocation failure GC in old space requested
[3966:0x55ad76a59a00]  1373301 ms: Mark-sweep 895.9 (1411.5) -> 895.8 (1355.5) MB, 826.3 / 0.0 ms  last resort GC in old space requested
[3966:0x55ad76a59a00]  1374124 ms: Mark-sweep 895.8 (1355.5) -> 895.8 (1344.0) MB, 823.4 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x27d3cf099099 <JSObject>
    1: set(this=0x160add029f09 <Map map = 0x100123f13a29>,0x2f7214e396c9 <String[36]\: \x9fsj\xbf8\xfbg<\xd9\xd6\x18\xe9\x01\xc2k\xee\xcc.\xe1h\xeeXj\xd3\xbf\xe0\x189\xc9\x1eH\xda\x01\x00\x00\x00>,0x2f7214e39709 <BufferItem map = 0x265c9dce0501>)
    2: commit [/home/jameson/Downloads/bcoin/node_modules/blru/lib/lru.js:~484] [pc=0x927044270c4](this=0x2f7214e399a1 <LRUBatch map = 0x265c9dcea4d9>)...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bcoin]
 2: 0x55ad75a4ef91 [bcoin]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [bcoin]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bcoin]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [bcoin]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [bcoin]
 7: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [bcoin]
 8: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [bcoin]
 9: 0x92703b840bd
Aborted

Am I using the max old space parameter correctly? Because the logs indicated that it was only using 1.5GB of js heap and 3GB of native heap even though I tried to give it 16GB.

@tuxcanfly

This comment has been minimized.

Copy link
Member

commented Nov 2, 2018

The argument --max-old-space-size needs to get passed down to node, which bcoin doesn't passthrough. So node is still using 1.5GB heap size.

An alternative way is to use environment variables to configure node options, so

export NODE_OPTIONS=--max_old_space_size=8192

in the environment before launching bcoin should work.

@bucko13

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

Yeah, also just a typo note, it should be underscores not dashes. A little more on the options here

@jlopp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 2, 2018

OK I used the environment variable to set the heap size to 16000 and it definitely made it bigger - the sync ran until block 286800 before crashing. In fact the bcoin process itself was using over 20GB of RAM when it finally blew heap.

export NODE_OPTIONS=--max-old-space-size=16000
bcoin --max-files 640 --cache-size 640 --checkpoints false --coin-cache 8000

[info] (net) Received 1 addrs (hosts=1000, peers=8) (194.15.231.236:8333).
[debug] (chain) Memory: rss=18717mb, js-heap=4194/7709mb native-heap=11008mb
[info] (chain) Block 00000000000000000f744d59b18215051ee7de538c0acc4903938d1845e3e383 (286800) added to chain (size=348915 txs=1062 time=160.999057).
[debug] (chain) Coin Cache: size=2000.564748764038mb, items=9121064.
[debug] (net) Status: time=2014-02-20T02:55:17Z height=286800 progress=52.18% orphans=8 active=368 target=419520339 peers=8

<--- Last few GCs --->

[8123:0x559daaee6bf0]  9909671 ms: Scavenge 4212.0 (7711.5) -> 4200.1 (7711.5) MB, 34.7 / 0.2 ms  allocation failure 
[8123:0x559daaee6bf0]  9909742 ms: Scavenge 4212.5 (7711.5) -> 4202.5 (7712.0) MB, 32.1 / 0.0 ms  allocation failure 
[8123:0x559daaee6bf0]  9910106 ms: Scavenge 4215.8 (7712.0) -> 4206.9 (7712.0) MB, 36.9 / 0.0 ms  allocation failure 
[8123:0x559daaee6bf0]  9910157 ms: Scavenge 4216.6 (7712.0) -> 4208.9 (7712.5) MB, 34.4 / 0.2 ms  allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3a17df599099 <JSObject>
    1: set(this=0x1ff479baa931 <Map map = 0x26bf08693a29>,0x3482f51049d9 <String[36]\: *\xdf?\xc4\x18\x96$\x13{\x93/\xb2mdC\xcc\xb6\xcbr\xec3\x90\x84\xc9\ri\xe4k\x18\x8d1\x8a\xd1\x01\x00\x00>,0x3482f5104a19 <BufferItem map = 0x26bf086dc619>)
    2: commit [/home/jameson/Downloads/bcoin/node_modules/blru/lib/lru.js:~378] [pc=0xef84c59819](this=0x1ff479ba7319 <LRU map = 0x25bbda142db9>)
    3: com...

FATAL ERROR: invalid table size Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bcoin]
 2: 0x559da9c38f91 [bcoin]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [bcoin]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bcoin]
 5: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Allocate(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [bcoin]
 6: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [bcoin]
 7: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [bcoin]
 8: 0xef848040bd
Aborted
@jlopp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 2, 2018

One possible issue which is still unclear is whether or not there is an 8GB heap limitation. The docs I have read said that it's theoretically unlimited, but that there could be quirks.

@braydonf

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

I think there is potential for risk of large pauses in mark-and-sweep garbage collection when the size of the heap grows, so there may be some additional side effects with increasing the old space size that can be undesirable.

@braydonf

This comment has been minimized.

Copy link
Contributor

commented Nov 2, 2018

There may be some opportunity to bind to C or possibly Node.js Buffers to have more fine to control over how memory handled. Would need to look more at the coin cache.

@jlopp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 3, 2018

FWIW I tried syncing with the following parameters as well (with the environment config on)

bcoin --max-files 640 --cache-size 640 --checkpoints false --coin-cache 4000
bcoin --checkpoints false --coin-cache 4000
bcoin --checkpoints false --coin-cache 3000
bcoin --checkpoints false --coin-cache 2000

And all of the above still resulted (eventually) in crashing with heap OOM errors.

So far the only set of options I've been able to run stably without crashing is

bcoin --checkpoints false --coin-cache 1000

@JasonCoombs

This comment has been minimized.

Copy link

commented Nov 6, 2018

Awesome work. Thanks for doing this, and to everyone who is offering input. 👍

@AlesTsurko

This comment has been minimized.

Copy link

commented Nov 16, 2018

Getting the "out of heap" too. My bcoin.conf is:

network: testnet

prefix: /data

use-workers: true

log-file: true
log-level: info

http-host: 0.0.0.0

spv: true

checkpoints: false
memory: false
mempool-memory: false
persistent: true
workers: true
listen: true

Note, I started to use bcoin-docker with may be updated version of bcoin. Previously I had no this error. Moreover I have two instances of bcoin (one for bitcoin, another — for bitcoin cash) on a server with 1 GB of RAM and they're working for months without such crashes.

@nodar-chkuaselidze

This comment has been minimized.

Copy link
Contributor

commented Nov 16, 2018

@AlesTsurko Was it bcoin or bcash that crashed ? (or both)

@AlesTsurko

This comment has been minimized.

Copy link

commented Nov 16, 2018

That was bcoin. But I haven't tried to use docker hub image for bcash... is there any, btw?

@kpjVideo

This comment has been minimized.

Copy link

commented Nov 18, 2018

I'm also having similar issues whilst the IBD (initial blockchain download) process is active.
I can typically get 30min-2hrs worth of runtime before the heap runs out of memory :(

OS: Windows 10 Professional
CPU: Intel Core i9-9900k
RAM: 32GB DDR4 @ 3200

Config:

node "%~dp0\node_modules\bcoin\bin\node" %* --max-old-space-size 8192 --prefix C:\BCoin-Data\data --coin-cache 3000 --index-tx --index-address

Recent Crashes:

<--- Last few GCs --->

[16700:0000014DD0C9F5F0]  4026566 ms: Mark-sweep 1205.0 (1479.9) -> 1204.8 (1483.4) MB, 1069.7 / 0.0 ms  allocation failure GC in old space requested
[16700:0000014DD0C9F5F0]  4027653 ms: Mark-sweep 1204.8 (1483.4) -> 1204.8 (1441.4) MB, 1086.5 / 0.0 ms  last resort GC in old space requested
[16700:0000014DD0C9F5F0]  4028750 ms: Mark-sweep 1204.8 (1441.4) -> 1204.8 (1436.4) MB, 1097.1 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000001CAC45A5879 <JSObject>
    1: delete(this=000002ACA23635E9 <Map map = 000002D0062048D9>,0000031CF4338E89 <String[64]: f0fedc5e46b5b7371d22ebc183cd89bd5616f3eaeee1d6110000000000000000>)
    2: removePeer [G:\BCoin-Node\node_modules\bcoin\lib\net\pool.js:~3033] [pc=0000002D6516AE22](this=000000F0557D7BA9 <EventEmitter map = 000002D006266599>,peer=000002ACA2363039 <EventEmitter map = 000002D006270519>)
    3: handleClo...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewFixedArray
 5: v8::internal::HashTable<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::IsKey
 6: v8::internal::Dictionary<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::AtPut
 7: v8::internal::compiler::RawMachineAssembler::MakeNode
 8: 0000002D648043C1
Press any key to continue . . .
<--- Last few GCs --->

[271252:000001E41FBBA650]  2395340 ms: Mark-sweep 1116.1 (1441.9) -> 1115.9 (1443.4) MB, 840.4 / 0.0 ms  allocation failure GC in old space requested
[271252:000001E41FBBA650]  2396187 ms: Mark-sweep 1115.9 (1443.4) -> 1115.9 (1387.4) MB, 847.7 / 0.0 ms  last resort GC in old space requested
[271252:000001E41FBBA650]  2397053 ms: Mark-sweep 1115.9 (1387.4) -> 1115.9 (1366.9) MB, 866.1 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 000002EE0D225879 <JSObject>
    1: set(this=0000023AB42758A9 <Map map = 000001B4A45048D9>,00000223DD95AD41 <String[65]: 06d96ae8b467f85912e216691b7fae8c1ab7b656b9e7a02631fdee705405c5ef0>,000002550FC9D599 <LRUItem map = 000001B4A4574509>)
    2: set [G:\BCoin-Node\node_modules\bcoin\node_modules\blru\lib\lru.js:~119] [pc=000000E7D06CA3D2](this=0000023AB425FB99 <LRU map = 000001289D3A4179>,key=00000223DD95AD41 <String[65]:...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewFixedArray
 5: v8::internal::HashTable<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::IsKey
 6: v8::internal::Dictionary<v8::internal::SeededNumberDictionary,v8::internal::SeededNumberDictionaryShape>::AtPut
 7: v8::internal::compiler::RawMachineAssembler::MakeNode
 8: 000000E7D05843C1
Press any key to continue . . .
@jlopp

This comment has been minimized.

Copy link
Contributor Author

commented Nov 19, 2018

In my experience if you assign more than 1 GB to the coin cache you're gonna have a bad time, regardless of how large you set the js heap.

@kpjVideo

This comment has been minimized.

Copy link

commented Nov 20, 2018

@jlopp Yeah, really in any configuration the same issue occurs on my end. 4 GB coincache, 1 GB coincache, 500MB coincache, unset coincache, all similar results :/

@chjj

This comment has been minimized.

Copy link
Member

commented Nov 27, 2018

Hmm, the memory usage estimation for UTXOs probably got a bit screwy for two reasons I think:

  1. The coin cache was kind of sloppily "ported" to the pertxout style of utxos (we used to cache Coins objects and now just cache individual coins under their own key -- pretty wasteful in terms of mem usage).
  2. The BufferMap refactor probably increased memory usage, just because it's storing an additional object along with a string key and buffer key for each Map entry.

Setting the coin cache too high will no doubt result in this. We could try to add some limits, but gauging v8 heap usage is difficult and may vary depending on os/arch.

Part of me actually doesn't really like the in-memory coin cache. I rarely see a perf increase on SSDs. We're probably duplicating a lot of work that leveldb already does with its built-in cache. The in-memory cache makes more sense if you do it bitcoin core style and write much larger batches every so often since the cache doubles as a batch itself, but we're just writing a batch on every block for simplicity (though we've had tons of experimentation with the core-style batch writes in the past).

Probably best to just fix the mem usage estimate for now.

@pinheadmz

This comment has been minimized.

Copy link
Member

commented Nov 27, 2018

@chjj is this related to the mempool size calculation?

#615
#481

@braydonf

This comment has been minimized.

Copy link
Contributor

commented Mar 21, 2019

Part of me actually doesn't really like the in-memory coin cache. I rarely see a perf increase on SSDs. We're probably duplicating a lot of work that leveldb already does with its built-in cache.

@chjj Indeed, in fact, it actually looks like it's slower with coin-cache, see discussion on: #672

It may be worth removing the coin-cache as it's not providing a performance improvement, and is causing confusion and issues with syncing. Multiple people have run into this issue during the IBD. I'll follow-up with more testing, but I think cache-size can be used instead, as it's passed to LevelDB.

tynes added a commit to tynes/hsd that referenced this issue Mar 29, 2019

blockchain: remove coin-cache
This PR removes the coin-cache from the blockchain.
It was determined that using the coin-cache actually
slows down the initial block download and can be
easily configured in such a way that it crashes Node.js.
This is a port of work done by braydonf on bcoin.

See relevant comment from jj: bcoin-org/bcoin#626 (comment)
See PR on bcoin here: bcoin-org/bcoin#736

@braydonf braydonf closed this in #736 Apr 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.