Nodeos crashed on replaying

[jeaimetu]

Action taken (what you did)

git fetch to download the latest source code
compile and install
kill current running nodes(previous version)
run nodeos (block pending error? I missed log of this)
kill nodes
run nodes with --replay it start to sync....
Expected result (what you hoped would happen)
after completion of replay, it runs normally.

Actual result (unexpected outcome)
nodes crashed with following message

2273625ms thread-0 fork_database.cpp:94 close ] states.size(): 1
2273625ms thread-0 controller.cpp:218 ~controller_impl ] db.revision(): 223865 head->block_num: 223865 blog.read_head()->block_num(): 1610970
2275368ms thread-0 main.cpp:99 main ] 13 St9exception: boost::interprocess::bad_alloc
boost::interprocess::bad_alloc:
{"what":"boost::interprocess::bad_alloc"}
thread-0 controller.cpp:944 finalize_block

{}
thread-0 controller.cpp:789 apply_block
rethrow
{}
thread-0 controller.cpp:800 push_block
rethrow
{"my->genesis_file.generic_string()":"/root/.local/share/eosio/nodeos/config/genesis.json"}
thread-0 chain_plugin.cpp:262 plugin_startup

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, etc)

meanwhile replaying blocks, I did cleos command as follows. But I do not think so following action is related to nodeos. It should be related to keosd.

root@ubuntu-s-4vcpu-8gb-nyc1-01:# cleos wallet open
"/usr/local/bin/keosd" launched
Opened: default
root@ubuntu-s-4vcpu-8gb-nyc1-01:# cleos wallet list
Wallets:
[
"default"
]

I tried this twice. details command is
"~/eos/build# nodeos -e -p eosio --plugin eosio::chain_api_plugin --plugin eosio::history_api_plugin --replay
"

same crash happended. with same error on replaying.

2568225ms thread-0 controller.cpp:736 start_block ] on block transaction failed, but shouldn't impact block generation, system contract needs update
2568225ms thread-0 controller.cpp:785 apply_block ] e.to_detail_string(): 13 St9exception: boost::interprocess::bad_alloc
boost::interprocess::bad_alloc:
{"what":"boost::interprocess::bad_alloc"}
thread-0 controller.cpp:944 finalize_block
2568225ms thread-0 fork_database.cpp:220 remove ] my->index.size(): 1
2568225ms thread-0 controller.cpp:800 push_block ] 13 St9exception: boost::interprocess::bad_alloc
boost::interprocess::bad_alloc:
{"what":"boost::interprocess::bad_alloc"}
thread-0 controller.cpp:944 finalize_block

{}
thread-0 controller.cpp:789 apply_block
2568225ms thread-0 chain_plugin.cpp:262 plugin_startup ] 13 St9exception: boost::interprocess::bad_alloc
boost::interprocess::bad_alloc:
{"what":"boost::interprocess::bad_alloc"}
thread-0 controller.cpp:944 finalize_block

{}
thread-0 controller.cpp:789 apply_block
rethrow
{}
thread-0 controller.cpp:800 push_block
2568225ms thread-0 chain_plugin.cpp:262 plugin_startup ] my->genesis_file.generic_string(): /root/.local/share/eosio/nodeos/config/genesis.json
2568226ms thread-0 fork_database.cpp:94 close ] states.size(): 1
2568226ms thread-0 controller.cpp:218 ~controller_impl ] db.revision(): 223865 head->block_num: 223865 blog.read_head()->block_num(): 1610970
2570042ms thread-0 main.cpp:99 main ] 13 St9exception: boost::interprocess::bad_alloc
boost::interprocess::bad_alloc:
{"what":"boost::interprocess::bad_alloc"}
thread-0 controller.cpp:944 finalize_block

{}
thread-0 controller.cpp:789 apply_block
rethrow
{}
thread-0 controller.cpp:800 push_block
rethrow
{"my->genesis_file.generic_string()":"/root/.local/share/eosio/nodeos/config/genesis.json"}
thread-0 chain_plugin.cpp:262 plugin_startup

I preserved this in my server and did not do any other things. if you need more information, then please let me know.

[bytemaster]

Your config file for shared memory and plugins is necessary. Shared memory file is probably too small or history plugin logging too much.

[jeaimetu]

Cool. Considering block number, that should be possible. But how can I configure that? Will it be supported or already supported through nodeos config.ini?

[andriantolie]

In your config.ini there is a field called chain-state-db-size-mb = 1024, this parameter controls the size of your shared memory file.
To control your history plugin logging, try not to put * on your filter-on = parameter in your config.ini but just specify the accounts that you want to keep track on. (If you haven't done anything with filter-on parameter previously, then history plugin should not log anything)

[jeaimetu]

I will try what you said and get back to here.

[ssinyagin]

@bytemaster but it's silly that you keep the account history index in RAM, and let it grow as the blockchain is growing. Maybe something like BerkeleyDB would be more appropriate. Account history is an important thing to provide from the nodes, so we should not let it grow in RAM without limits. BerkeleyDB is around for over 20 years, and it has a limited RAM footprint.

[ssinyagin]

my apologies, it appears to be the size of on-disk shared memory. The daemon even manages to do indexing within 1GB RAM. Quite clever :)

[mubashar-activemobile]

i am having the same problem.
my chain-state-db-size-mb = 102400
yet the node crashes with the following error

2018-08-06T10:02:08.927 thread-0 controller.cpp:153 emit ] bad alloc
2018-08-06T10:02:14.298 thread-0 main.cpp:128 main ] bad alloc

[andriantolie]

How much is your actually RAM in your machine? And do you enable history plugin? Even though you can put a number higher than your actual RAM, how it is being handled internally is dependent on the operating system (whether it's going to use swapfile or crash)