3060101 database_guard_exception: Database usage is at unsafe levels

[cppfuns]

root@iZj6c7toj4996n194z4h2pZ:~# nodeos --data-dir /mnt/data/data/ --hard-replay-blockchain
2018-08-08T07:13:19.131 thread-0 chain_plugin.cpp:271 plugin_initialize ] initializing chain plugin
2018-08-08T07:13:19.132 thread-0 chain_plugin.cpp:414 plugin_initialize ] Hard replay requested: deleting state database
2018-08-08T07:13:19.520 thread-0 block_log.cpp:325 repair_log ] Recovering Block Log...
2018-08-08T07:13:19.520 thread-0 block_log.cpp:345 repair_log ] Moved existing blocks directory to backup location: '/mnt/data/data/blocks-2018-08-08T07:13:19.520'
2018-08-08T07:13:19.521 thread-0 block_log.cpp:350 repair_log ] Reconstructing '/mnt/data/data/blocks/blocks.log' from backed up block log

2018-08-08T07:16:03.886 thread-0 block_log.cpp:445 repair_log ] Recovered only up to block number 9570628. The block 9570629 could not be deserialized from the block log due to error:
Assert Exception
2018-08-08T07:16:06.499 thread-0 block_log.cpp:455 repair_log ] Data at tail end of block log which should contain the (incomplete) serialization of block 9570629 has been written out to '/mnt/data/data/blocks/blocks-bad-tail-2018-08-08T07:13:19.520.log'.
2018-08-08T07:16:06.746 thread-0 chain_plugin.cpp:681 recover_reversible_b ] Reconstructing '/mnt/data/data/blocks/reversible' from backed up reversible directory
2018-08-08T07:16:08.887 thread-0 chain_plugin.cpp:733 recover_reversible_b ] Recovered 343 blocks from reversible block database: blocks 10012920 to 10013262
2018-08-08T07:16:12.010 thread-0 block_log.cpp:124 open ] Log is nonempty
2018-08-08T07:16:12.010 thread-0 block_log.cpp:159 open ] Index is empty
2018-08-08T07:16:12.010 thread-0 block_log.cpp:298 construct_index ] Reconstructing Block Log Index...
2018-08-08T07:17:23.863 thread-0 http_plugin.cpp:344 plugin_initialize ] configured http to listen on 0.0.0.0:8888
2018-08-08T07:17:23.865 thread-0 history_plugin.cpp:275 plugin_initialize ] --filter-on * enabled. This can fill shared_mem, causing nodeos to stop.
2018-08-08T07:17:23.867 thread-0 net_plugin.cpp:2919 plugin_initialize ] Initialize net plugin
2018-08-08T07:17:23.868 thread-0 net_plugin.cpp:2944 plugin_initialize ] host: 0.0.0.0 port: 9876
2018-08-08T07:17:23.870 thread-0 net_plugin.cpp:3014 plugin_initialize ] my node_id is 74f8379ab3548acd26575f44e35876213f02982fe42edcca6725e70c643a0e33
2018-08-08T07:17:23.876 thread-0 http_plugin.cpp:299 operator() ] configured http with Access-Control-Allow-Origin: *
2018-08-08T07:17:23.877 thread-0 main.cpp:105 main ] nodeos version 8777d8d
2018-08-08T07:17:23.877 thread-0 main.cpp:106 main ] eosio root is /root/.local/share
2018-08-08T07:17:23.892 thread-0 controller.cpp:1256 startup ] No head block in fork db, perhaps we need to replay
2018-08-08T07:17:23.892 thread-0 controller.cpp:319 initialize_fork_db ] Initializing new blockchain with genesis state
2018-08-08T07:17:23.929 thread-0 controller.cpp:215 init ] existing block log, attempting to replay 9570628 blocks
2018-08-08T07:43:25.133 thread-0 wasm_interface.cpp:929 eosio_assert ] message: refund is not available yet
2018-08-08T07:47:21.970 thread-0 controller.cpp:612 push_scheduled_trans ] 3080004 tx_cpu_usage_exceeded: Transaction exceeded the current CPU usage limit imposed on the transaction
billed CPU time (9135 us) is greater than the maximum billable CPU time for the transaction (8751 us)
{"billed":9135,"billable":8751}
thread-0 transaction_context.cpp:361 validate_cpu_usage_to_bill
2018-08-08T07:48:02.716 thread-0 controller.cpp:612 push_scheduled_trans ] 3080004 tx_cpu_usage_exceeded: Transaction exceeded the current CPU usage limit imposed on the transaction
billed CPU time (11027 us) is greater than the maximum billable CPU time for the transaction (8752 us)
{"billed":11027,"billable":8752}
thread-0 transaction_context.cpp:361 validate_cpu_usage_to_bill
2018-08-08T07:48:21.075 thread-0 wasm_interface.cpp:929 eosio_assert ] message: refund is not available yet
2018-08-08T07:48:21.078 thread-0 controller.cpp:161 emit ] signal handler threw exception
2018-08-08T07:48:27.088 thread-0 wasm_interface.cpp:929 eosio_assert ] message: refund request not found
2018-08-08T07:48:27.088 thread-0 controller.cpp:161 emit ] signal handler threw exception
2018-08-08T07:48:27.340 thread-0 wasm_interface.cpp:929 eosio_assert ] message: refund is not available yet
2018-08-08T07:48:27.340 thread-0 controller.cpp:161 emit ] signal handler threw exception
2018-08-08T07:48:41.244 thread-0 controller.cpp:612 push_scheduled_trans ] 3080004 tx_cpu_usage_exceeded: Transaction exceeded the current CPU usage limit imposed on the transaction
billed CPU time (19455 us) is greater than the maximum billable CPU time for the transaction (17510 us)
{"billed":19455,"billable":17510}
thread-0 transaction_context.cpp:361 validate_cpu_usage_to_bill
2018-08-08T07:48:45.847 thread-0 wasm_interface.cpp:929 eosio_assert ] message: refund request not found
2018-08-08T07:48:45.847 thread-0 controller.cpp:161 emit ] signal handler threw exception
2018-08-08T07:48:46.845 thread-0 wasm_interface.cpp:929 eosio_assert ] message: refund request not found
2018-08-08T07:48:46.846 thread-0 controller.cpp:161 emit ] signal handler threw exception
2018-08-08T07:49:46.408 thread-0 controller.cpp:612 push_scheduled_trans ] 3080004 tx_cpu_usage_exceeded: Transaction exceeded the current CPU usage limit imposed on the transaction
billed CPU time (1380 us) is greater than the maximum billable CPU time for the transaction (817 us)
{"billed":1380,"billable":817}
thread-0 transaction_context.cpp:361 validate_cpu_usage_to_bill
2018-08-08T07:50:14.561 thread-0 controller.cpp:612 push_scheduled_trans ] 3080004 tx_cpu_usage_exceeded: Transaction exceeded the current CPU usage limit imposed on the transaction
billed CPU time (1321 us) is greater than the maximum billable CPU time for the transaction (743 us)
{"billed":1321,"billable":743}
thread-0 transaction_context.cpp:361 validate_cpu_usage_to_bill
2018-08-08T07:50:48.420 thread-0 chain_plugin.cpp:853 log_guard_exception ] Database has reached an unsafe level of usage, shutting down to avoid corrupting the database. Please increase the value set for "chain-state-db-size-mb" and restart the process!
2018-08-08T07:50:48.420 thread-0 chain_plugin.cpp:859 log_guard_exception ] Details: 3060101 database_guard_exception: Database usage is at unsafe levels
database free: 134217584, guard size: 134217728
{"f":134217584,"g":134217728}
thread-0 controller.cpp:1627 validate_db_available_size
2018-08-08T07:50:49.091 thread-0 main.cpp:125 main ] 3060101 database_guard_exception: Database usage is at unsafe levels
database free: 134217584, guard size: 134217728
{"f":134217584,"g":134217728}
thread-0 controller.cpp:1627 validate_db_available_size

{}
thread-0  chain_plugin.cpp:603 plugin_startup

[taokayan]

Looks like you're syncing up with the mainnet. What's your database size and what's your hardware config?

[cppfuns]

database size 10GB+
CPU Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz
memory 32GiB
disk 1T SSD

[tqknight1984]

did you solve this problem?

[cppfuns]

No solution has been found yet.

[wanderingbort]

The relevant suggestion got buried a little bit in your log:

Database has reached an unsafe level of usage, shutting down to avoid corrupting the database. Please increase the value set for "chain-state-db-size-mb" and restart the process!

what is the value of chain-state-db-sze_mb in your config.ini?
do you have the history_plugin enabled?
if so what is your filter-on setting?

[cppfuns]

chain-state-db-sze_mb 2500 +
history_plugin enabled
filter-on *

[cppfuns]

And I don't know how to use filter-on to filter out the parts I want, such as receiving only transactions.

[xlk3099]

I got the same error, waiting for some useful feedback as well.

[zhousi666]

@cppfuns Did you solve this problem? I got the same error,too.

[bboyHan]

I got the same error, waiting for some useful feedback as well.
hope soon.

[Drug0j]

I was facing the same issue. I'm running my node in a docker container and I was able to resolve this error by going to my config.ini file and increasing the chain-state-db-size-mb (default 1024MB), which is the parameter that specifies the maximum size of the shared_memory database for storing the state. Remember to configure it to be 4000 mb than your machine's memory. Hope this helps!

[sensay-nelson]

I set chain-state-db-size-mb to 28000, running on a machine with 32GB of memory, and am still seeing this error. Running v1.2.5 of the eos.io software. It's so frustrating not being able to find documentation for what this parameter represents, what are minimum requirements, and how to configure it correctly.

[hershkoy]

Also seeing the issue. Also trying the reply chain from docker, using v1.2.5.
Disk is 200Gb, but only 28% is utilized. RAM is 4GB.
I set chain-state-db-size-mb = 65536 in config, don't know if this value is good.
history plugin is enabled in config.ini

[lcgogo]

met the same issue,
chain-state-db-sze_mb 8196
history_plugin enabled
filter-on = "*"

[jgiszczak]

With history_plugin enabled and a filter-on of "*", the chain state on a node synced with mainnet is absolutely massive. Impractically massive, which is why history_plugin is deprecated. If you wish to retain a completely unfiltered history, you should be using mongo_db_plugin or sql_db_plugin.

Without the history plugin, chain state on mainnet is currently ~1.8GiB. The default size is 1GiB, which is no longer large enough to accommodate even the minimal state size of mainnet.

Out of an abundance of caution, producer nodes are using chain-state-db-size-mb = 65536.

[godmar]

I'm seeing this on a single node set up for testing, used for nothing but the tutorial and a minor amount of experimentation. After about 8 days of running it shut down after 1,458,965 blocks with:

info  2019-11-20T16:24:39.000 nodeos    producer_plugin.cpp:1914      produce_block        ] Produced block 0016431698207891... #1458966 @ 2019-11-20T16:24:39.000 signed by eosio [trxs: 0, lib: 1458965, confirmed: 0]
error 2019-11-20T16:24:39.500 nodeos    chain_plugin.cpp:1245         log_guard_exception  ] Database has reached an unsafe level of usage, shutting down to avoid corrupting the database.  Please increase the value set for "chain-state-db-size-mb" and restart the process!
error 2019-11-20T16:24:39.500 nodeos    producer_plugin.cpp:1755      schedule_production_ ] Failed to start a pending block, will try again later
info  2019-11-20T16:24:39.548 nodeos    main.cpp:141                  main                 ] nodeos successfully exiting

I installed eosi 1.8.6 (the most recent release) as per release instructions.

I see that it recommends changing a configuration option chain-state-db-size-mb. One issue is that the Ubuntu package didn't come with a config file:

$ dpkg -L eosio
/.
/usr
/usr/bin
/usr/opt
/usr/opt/eosio
/usr/opt/eosio/1.8.6
/usr/opt/eosio/1.8.6/bin
/usr/opt/eosio/1.8.6/bin/cleos
/usr/opt/eosio/1.8.6/bin/keosd
/usr/opt/eosio/1.8.6/bin/nodeos
/usr/opt/eosio/1.8.6/licenses
/usr/opt/eosio/1.8.6/licenses/LICENSE
/usr/opt/eosio/1.8.6/licenses/LICENSE.go
/usr/opt/eosio/1.8.6/licenses/LICENSE.secp256k1
/usr/opt/eosio/1.8.6/licenses/LICENSE.softfloat
/usr/opt/eosio/1.8.6/licenses/LICENSE.wabt
/usr/opt/eosio/1.8.6/licenses/LICENSE.wavm
/usr/opt/eosio/1.8.6/licenses/LICENSE.yubihsm
/usr/opt/eosio/1.8.6/licenses/eosio
/usr/bin/cleos
/usr/bin/keosd
/usr/bin/nodeos

so I am assuming default values of all config parameters are being used.

With regular Unix-based software, I would now read the man page to learn where the config file is located (presumably somewhere in /etc?), but unfortunately, this package does not seem to include man pages. Googling shows a web page in Korean. Running locate config | grep nodeos shows a config in ~/.local/share/eosio/nodeos/config/config.ini

Changing the value in this file to 65536, as recommended in this thread, allows me to start nodeos again.

Possible suggestions include:

• use standard conventions to help system administrators learn about the configuration of your software

• include Unix man pages that describe config file parameters.

• perhaps make the default configuration not such that it stops after 8 days of minuscule use.