Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Out of memory in long term runs #1475

Open
zhenwusw opened this issue Sep 11, 2017 · 9 comments
Open

Out of memory in long term runs #1475

zhenwusw opened this issue Sep 11, 2017 · 9 comments

Comments

@zhenwusw
Copy link

zhenwusw commented Sep 11, 2017 •
edited
Loading

Hi,

Not sure if it's a duplicated issue with bitpay/bitcore-node#412, but it happens on bitcore@5.0.0-beta.8, any ideas?

<--- Last few GCs --->

[21475:0x3d14c30]    67692 ms: Mark-sweep 1405.7 (1461.9) -> 1405.6 (1433.9) MB, 1088.7 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1089 ms) last resort
[21475:0x3d14c30]    68789 ms: Mark-sweep 1405.6 (1433.9) -> 1405.6 (1433.9) MB, 1096.7 / 0.0 ms  last resort


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x31bc6ff1cef1 <JSObject>
    1: set [/home/deploy/.nvm/versions/node/v8.4.0/lib/node_modules/bitcore/node_modules/bitcore-node/lib/utils.js:~83] [pc=0x293ee3031feb](this=0x3d8d5be7c841 <SimpleMap map = 0x2e23834c4fc1>,key=0x24404b012be9 <String[64]: 0000000008803db075ddce44df99d36367ee7edc80cc96238bc9449aa92f1d39>,value=0x2538671bbb99 <Object map = 0x2e2383492741>,pos=5461)
    2: /* anonymous */ [/home/deploy/.nvm/vers...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [bitcored]
 2: 0x13740dc [bitcored]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [bitcored]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [bitcored]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [bitcored]
 6: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::New(v8::internal::Isolate*, int, v8::internal::MinimumCapacity, v8::internal::PretenureFlag) [bitcored]
 7: v8::internal::HashTable<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::Handle<v8::internal::Name>, v8::internal::PretenureFlag) [bitcored]
 8: v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::EnsureCapacity(v8::internal::Handle<v8::internal::NameDictionary>, int, v8::internal::Handle<v8::internal::Name>) [bitcored]
 9: v8::internal::Dictionary<v8::internal::NameDictionary, v8::internal::NameDictionaryShape, v8::internal::Handle<v8::internal::Name> >::Add(v8::internal::Handle<v8::internal::NameDictionary>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyDetails, int*) [bitcored]
10: v8::internal::LookupIterator::ApplyTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>) [bitcored]
11: v8::internal::Object::AddDataProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::ShouldThrow, v8::internal::Object::StoreFromKeyed) [bitcored]
12: v8::internal::Object::SetProperty(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [bitcored]
13: v8::internal::Runtime_SetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [bitcored]
14: 0x293ee2e840dd
Aborted
@pmosse
Copy link

pmosse commented Sep 12, 2017

HI! I am having the same issue (I am using the testnet). Did you manage to solve it? Do you see any blocks in the Insight UI? Thanks!

@chrisrico
Copy link

Are you fully synced? I had an issue in the ~380k block height area where I kept running out of memory. I had to increase the Node.js heap size like this:

node --max_old_space_size=4096 path/to/bitcored -c path/to/config

@osagga
Copy link
Contributor

osagga commented Jul 29, 2019

can confirm that this problem still happens in the latest bitcore-node from master.

I have two nodes running completley sperate from each other, started syncning at the same time and been running for the same amount of time.

The first one crashed at 18:03 UTC and the other one at 18:09 UTC (so about 6 minutes apart)

The crash message on the first one is:

<--- Last few GCs --->

[14365:0x3496ab0] 531242311 ms: Mark-sweep 1318.2 (1425.2) -> 1317.2 (1425.2) MB, 1636.2 / 0.0 ms  (average mu = 0.374, current mu = 0.380) allocation failure scavenge might not succeed
[14365:0x3496ab0] 531244197 ms: Mark-sweep 1318.2 (1425.2) -> 1317.2 (1425.2) MB, 1624.5 / 0.0 ms  (average mu = 0.275, current mu = 0.139) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x16494fb5be1d]
    1: StubFrame [pc: 0x16495050544d]
Security context: 0x0f80f059e6e9 <JSObject>
    2: authenticateStragglers(aka authenticateStragglers) [0x2b9a38b07991] [/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/connection/pool.js:1] [bytecode=0x2b298f572bd9 offset=0](this=0x24e485b026f1 <undefined>,self=0x0058cbb
95071 <Pool map = 0x324eeb277f79>,connection=0x0e55726ac9e9 <Connecti...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8f9d10 node::Abort() [node]
 2: 0x8f9d5c  [node]
 3: 0xaffd0e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xafff44 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xef4152  [node]
 6: 0xef4258 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xf00332 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xf00c64 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf038d1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xeccd54 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x116cede v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x16494fb5be1d 
sh: line 1: 14365 Aborted                 node build/src/server.js
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! bitcore-node@8.3.4 start: `npm run tsc && node build/src/server.js`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the bitcore-node@8.3.4 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bitcore/.npm/_logs/2019-07-29T18_03_40_741Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! @8.1.0 node: `cd packages/bitcore-node && npm start`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the @8.1.0 node script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bitcore/.npm/_logs/2019-07-29T18_03_40_763Z-debug.log

The crash message on the second one is:

<--- Last few GCs --->

[14365:0x3496ab0] 531242311 ms: Mark-sweep 1318.2 (1425.2) -> 1317.2 (1425.2) MB, 1636.2 / 0.0 ms  (average mu = 0.374, current mu = 0.380) allocation failure scavenge might not succeed
[14365:0x3496ab0] 531244197 ms: Mark-sweep 1318.2 (1425.2) -> 1317.2 (1425.2) MB, 1624.5 / 0.0 ms  (average mu = 0.275, current mu = 0.139) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x16494fb5be1d]
    1: StubFrame [pc: 0x16495050544d]
Security context: 0x0f80f059e6e9 <JSObject>
    2: authenticateStragglers(aka authenticateStragglers) [0x2b9a38b07991] [/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/connection/pool.js:1] [bytecode=0x2b298f572bd9 offset=0](this=0x24e485b026f1 <undefined>,self=0x0058cbb
95071 <Pool map = 0x324eeb277f79>,connection=0x0e55726ac9e9 <Connecti...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8f9d10 node::Abort() [node]
 2: 0x8f9d5c  [node]
 3: 0xaffd0e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xafff44 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xef4152  [node]
 6: 0xef4258 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xf00332 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xf00c64 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xf038d1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xeccd54 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [node]
11: 0x116cede v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [node]
12: 0x16494fb5be1d 
sh: line 1: 14365 Aborted                 node build/src/server.js
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! bitcore-node@8.3.4 start: `npm run tsc && node build/src/server.js`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the bitcore-node@8.3.4 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bitcore/.npm/_logs/2019-07-29T18_03_40_741Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! @8.1.0 node: `cd packages/bitcore-node && npm start`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the @8.1.0 node script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/bitcore/.npm/_logs/2019-07-29T18_03_40_763Z-debug.log

I've seen this error before after running the node for a while (about 3 days), and I'm not fully sure if this is due to a memory leak, or weirdly formed transaction on the blockchain. Also note that this node is running on testnet.

Any thoughts to what could be causing this memory leak? or is this crash expected?

@micahriggan

@micahriggan
Copy link
Contributor

testnet btc?

@osagga
Copy link
Contributor

osagga commented Jul 30, 2019 •
edited
Loading

@micahriggan it's testnet, but connecting to btc, bch and ltc fullnodes at the same time

@micahriggan
Copy link
Contributor

I think this is probably related to 32 MB blocks. I've seen really high memory usage from that.

My guess would be from https://github.com/bitpay/bitcore/blob/master/packages/bitcore-node/src/models/transaction.ts#L142

These promise.all's are probably too large for 32MB

@osagga
Copy link
Contributor

osagga commented Aug 5, 2019

@micahriggan I see, are there any ways to fix this? Or should I just increase the default memory limit? I would love to have bitcore running out of the box without having to modify the launch memory limit.

@micahriggan
Copy link
Contributor

We've been messing with this recently. I think the best solution is to use CHAIN=BCH NETWORK=testnet npm run p2p and run BCH testnet on it's own process

@osagga
Copy link
Contributor

osagga commented Sep 10, 2019

@micahriggan Hey, so recently the BCH testnet network had big blocks and bitcore seems to be stuck processing them with the following error:

error: 2019-09-10 16:58:43.188 UTC | Error syncing | Chain: BCH | Network: testnet RangeError [ERR_BUFFER_OUT_OF_BOUNDS]: Attempt to write outside buffer bounds
    at Buffer.write (buffer.js:922:13)
    at serializeString (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:34:14)
    at serializeInto (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:709:17)
    at serializeObject (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:347:18)
    at serializeInto (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:937:17)
    at serializeObject (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:347:18)
    at serializeInto (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:937:17)
    at serializeObject (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:347:18)
    at serializeInto (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/parser/serializer.js:937:17)
    at BSON.serialize (/home/bitcore/bitcore/packages/bitcore-node/node_modules/bson/lib/bson/bson.js:63:28)
    at Query.toBin (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/connection/commands.js:144:25)
    at serializeCommands (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/connection/pool.js:1044:43)
    at Pool.write (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/connection/pool.js:1260:3)
    at Cursor._find (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/cursor.js:326:22)
    at nextFunction (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/cursor.js:673:10)
    at Cursor.next (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb-core/lib/cursor.js:824:3)
    at Cursor._next (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb/lib/cursor.js:211:36)
    at fetchDocs (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb/lib/operations/cursor_ops.js:211:12)
    at toArray (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb/lib/operations/cursor_ops.js:241:3)
    at /home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb/lib/utils.js:437:24
    at new Promise (<anonymous>)
    at executeOperation (/home/bitcore/bitcore/packages/bitcore-node/node_modules/mongodb/lib/utils.js:432:10)

Do you think this could be fixed with a code change? do you have any pointers where I could dig in to look?

I tried increasing the heap size (memory) and that didn't fix it still.

@osagga osagga mentioned this issue Sep 11, 2019
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@zhenwusw @micahriggan @chrisrico @osagga @pmosse