Uncaught exception during sync with mainnet

[n3b]

1678107ms thread-0   producer_plugin.cpp:294       on_incoming_block    ] Received block 360000f9f93f5149... #4418000 @ 2018-07-06T06:06:11.000 signed by eosriobrazil [trxs: 2, lib: 4417665, conf: 0, latency: 192107107 ms]
1798989ms thread-0   producer_plugin.cpp:294       on_incoming_block    ] Received block 3ab19335dd906cf8... #4419000 @ 2018-07-06T06:14:37.000 signed by eosstorebest [trxs: 1, lib: 4418673, conf: 0, latency: 191721989 ms]
1905874ms thread-0   producer_plugin.cpp:294       on_incoming_block    ] Received block 8e1fb664cacd0c30... #4420000 @ 2018-07-06T06:22:57.000 signed by eosriobrazil [trxs: 1, lib: 4419669, conf: 0, latency: 191328874 ms]
1912728ms thread-0   wasm_interface.cpp:929        eosio_assert         ] message: refund request not found 
1912728ms thread-0   controller.cpp:589            push_scheduled_trans ] 3050003 eosio_assert_message_exception: eosio_assert_message assertion failure
assertion failure with message: refund request not found
    {"s":"refund request not found"}
    thread-0  wasm_interface.cpp:930 eosio_assert

    {"_pending_console_output.str()":""}
    thread-0  apply_context.cpp:62 exec_one
2056623ms thread-0   producer_plugin.cpp:294       on_incoming_block    ] Received block a5a4c5fa71c40f3b... #4421000 @ 2018-07-06T06:31:18.000 signed by eosriobrazil [trxs: 21, lib: 4420676, conf: 240, latency: 190978623 ms]
2284650ms thread-0   producer_plugin.cpp:294       on_incoming_block    ] Received block 9dfc5ac53a9e3ea1... #4422000 @ 2018-07-06T06:39:38.000 signed by eosnewyorkio [trxs: 17, lib: 4421671, conf: 0, latency: 190706650 ms]
libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<std::logic_error> >: Could not restore object, most likely a uniqueness constraint was violated
Abort trap: 6

Is there any workaround to avoid sync/hard-replay of 4.5kk blocks again??

[hzqlmm]

I meet same problem too.

[Ampakinetic]

Same problem here, is there a workaround?

[Ampakinetic]

Still unable to sync with Main net, it consistently blows up on block 1027926 with a cascade of errors that look like:

527684ms thread-0 controller.cpp:855 push_block ] 10 assert_exception: Assert Exception
prior != by_id_idx.end(): unlinkable block
{"id":"000fb5cc67df195149b87ee7ce37ea10d4309399ae43b28085fb8e9d36eba9e4","previous":"000fb5cb40d4a99302614ab013eb19ce7424f70626b7149ca3d4c0690b6f5d47"}
thread-0 fork_database.cpp:152 add
527684ms thread-0 producer_plugin.cpp:274 on_incoming_block ] 10 assert_exception: Assert Exception
prior != by_id_idx.end(): unlinkable block
{"id":"000fb5cc67df195149b87ee7ce37ea10d4309399ae43b28085fb8e9d36eba9e4","previous":"000fb5cb40d4a99302614ab013eb19ce7424f70626b7149ca3d4c0690b6f5d47"}
thread-0 fork_database.cpp:152 add
rethrow
{}
thread-0 controller.cpp:855 push_block

[n3b]

Actually there is a workaround, you can stop the nodeos and make a backup of blocks and state. But it only will help you to start sync from where you've made a backup.

However, the sync is always being dropped on a block 4422156 (or maybe 4422157). It is just impossible to pass through that with current release.

[heifner]

I have not been able to reproduce this with release/1.1 branch. Can you give more information? What do you have chain-state-db-size-mb and reversible-blocks-db-size-mb set at? Do you have any other non-default settings? What platform? What exact version of code?

[n3b]

@heifner it was 1.0.8. Just updated to 1.1 and it seems everything is fine now. Thanks!