v3.1.0 Archive Node Sync Failure: "event.loop0' panicked at 'Propagating verifier thread panic on shutdown: Any'"

[ymonye]

• OpenEthereum version: 3.1.0
• Operating system: Linux CentOS 7
• Installation: Built from source
• Fully synchronized: Fails to even start sync
• Network: Ethereum Mainnet
• Restarted: Yes

Hi. I've been using OpenEthereum 3.0 and prior versions for the past year. Upon installing v3.1 on a new machine with the same hardware / software specs (CentOS 7), I attempted to start a new sync using the previous archive node commands below:

openethereum --pruning=archive --fat-db=on --tracing=on

This resulted in the following error:

Thread 'event.loop0' panicked at 'Propagating verifier thread panic on shutdown: Any', /openethereum/ethcore/src/verification/queue/mod.rs:853

Running a regular sync, the openethereum command without any arguments, works without issue.

Rust Verstion:
rustc 1.48.0 (7eac88abb 2020-11-16)

Below is the full trace (pastebin keeps deleting my paste):

openethereum --pruning=archive --fat-db=on --tracing=on
2020-12-03 11:44:31 UTC Starting OpenEthereum/v3.1.0-rc1-stable-ca3a82f-20201130/x86_64-linux-gnu/rustc1.48.0
2020-12-03 11:44:31 UTC Keys path /root/.local/share/openethereum/keys/ethereum
2020-12-03 11:44:31 UTC DB path /root/.local/share/openethereum/chains/ethereum/db/906a34e69aec8c0d
2020-12-03 11:44:31 UTC State DB configuration: archive +Fat +Trace
2020-12-03 11:44:31 UTC Operating mode: active
2020-12-03 11:44:31 UTC Warning: Warp Sync is disabled because Fat DB is turned on.
2020-12-03 11:44:31 UTC Configured for Ethereum using Ethash engine
2020-12-03 11:44:31 UTC Listening for new connections on 127.0.0.1:8546.
2020-12-03 11:44:31 UTC Updated conversion rate to Ξ1 = US$612.72 (7771747.5 wei/gas)
2020-12-03 11:44:33 UTC Panic occured, see stderr for details

====================

stack backtrace:
0: panic_hook::set_with::{{closure}}
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:581
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:484
3: std::sys_common::backtrace::__rust_end_short_backtrace
at library/std/src/sys_common/backtrace.rs:153
4: rust_begin_unwind
at library/std/src/panicking.rs:483
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:85
6: core::panicking::panic
at library/core/src/panicking.rs:50
7: linked_hash_map::LinkedHashMap<K,V,S>::insert
8: lru_cache::LruCache<K,V,S>::insert
9: ethcore::state_db::StateDB::sync_cache
10: ethcore::client::client::Importer::import_verified_blocks
11: <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandlerethcore::client::io_message::ClientIoMessage>::message
12: ethcore_io::service_mio::IoChannel::send_sync
13: std::sys_common::backtrace::__rust_begin_short_backtrace
14: core::ops::function::FnOnce::call_once{{vtable.shim}}
15: <alloc::boxed::Box as core::ops::function::FnOnce>::call_once
at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
<alloc::boxed::Box as core::ops::function::FnOnce>::call_once
at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
std::sys::unix::thread::Thread::new::thread_start
at library/std/src/sys/unix/thread.rs:87
16: start_thread
17: __clone

Thread 'Verifier #3' panicked at 'attempted to leave type linked_hash_map::Node<ethereum_types::H160, std::option::Option<state::account::Account>> uninitialized, which is invalid', /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/mem/mod.rs:658

This is a bug. Please report it at:

https://github.com/openethereum/openethereum/issues/new

2020-12-03 11:44:33 UTC Finishing work, please wait...
2020-12-03 11:44:33 UTC Block import failed for #2 (0xb495…98c9): Parent not found (0x88e9…6cb6)
2020-12-03 11:44:33 UTC
Bad block detected: Error(Msg("Parent not found"), State { next_error: None, backtrace: InternalBacktrace { backtrace: None } })
RLP: f9021df90218a088e96d4537bea4d9c05d12549907b32561d3bf31f45aae734cdc119f13406cb6a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794dd2f1e6e498202e86d8f5442af596580a4f03c2ca04943d941637411107494da9ec8bc04359d731bfd08b72b4d0edcbd4cd2ecb341a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421b90100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008503ff00100002821388808455ba4241a0476574682f76312e302e302d30636463373634372f6c696e75782f676f312e34a02f0790c5aa31ab94195e1f6443d645af5b75c46c04fbf9911711198a0ce8fdda88b853fa261a86aa9ec0c0
Header: Header { parent_hash: 0x88e96d4537bea4d9c05d12549907b32561d3bf31f45aae734cdc119f13406cb6, timestamp: 1438270017, number: 2, author: 0xdd2f1e6e498202e86d8f5442af596580a4f03c2c, transactions_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, uncles_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, extra_data: [71, 101, 116, 104, 47, 118, 49, 46, 48, 46, 48, 45, 48, 99, 100, 99, 55, 54, 52, 55, 47, 108, 105, 110, 117, 120, 47, 103, 111, 49, 46, 52], state_root: 0x4943d941637411107494da9ec8bc04359d731bfd08b72b4d0edcbd4cd2ecb341, receipts_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, log_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, gas_used: 0, gas_limit: 5000, difficulty: 17163096064, seal: [[160, 47, 7, 144, 197, 170, 49, 171, 148, 25, 94, 31, 100, 67, 214, 69, 175, 91, 117, 196, 108, 4, 251, 249, 145, 23, 17, 25, 138, 12, 232, 253, 218], [136, 184, 83, 250, 38, 26, 134, 170, 158]], hash: Some(0xb495a1d7e6663152ae92708da4843337b958146015a2802f4193a410044698c9) }
Uncles:
Transactions:

2020-12-03 11:44:33 UTC Panic occured, see stderr for details

====================

stack backtrace:
0: panic_hook::set_with::{{closure}}
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:581
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:484
3: std::sys_common::backtrace::__rust_end_short_backtrace
at library/std/src/sys_common/backtrace.rs:153
4: rust_begin_unwind
at library/std/src/panicking.rs:483
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:85
6: core::panicking::panic
at library/core/src/panicking.rs:50
7: linked_hash_map::LinkedHashMap<K,V,S>::insert
8: lru_cache::LruCache<K,V,S>::insert
9: memory_cache::MemoryLruCache<K,V>::insert
10: ethcore::client::bad_blocks::BadBlocks::report
11: ethcore::client::client::Importer::import_verified_blocks
12: <ethcore_service::service::ClientIoHandler as ethcore_io::IoHandlerethcore::client::io_message::ClientIoMessage>::message
13: ethcore_io::service_mio::IoChannel::send_sync
14: std::sys_common::backtrace::__rust_begin_short_backtrace
15: core::ops::function::FnOnce::call_once{{vtable.shim}}
16: <alloc::boxed::Box as core::ops::function::FnOnce>::call_once
at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
<alloc::boxed::Box as core::ops::function::FnOnce>::call_once
at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
std::sys::unix::thread::Thread::new::thread_start
at library/std/src/sys/unix/thread.rs:87
17: start_thread
18: __clone

Thread 'Verifier #0' panicked at 'attempted to leave type linked_hash_map::Node<ethereum_types::H256, (verification::queue::kind::blocks::Unverified, std::string::String)> uninitialized, which is invalid', /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/mem/mod.rs:658

This is a bug. Please report it at:

https://github.com/openethereum/openethereum/issues/new

2020-12-03 11:44:36 UTC Panic occured, see stderr for details

====================

stack backtrace:
0: panic_hook::set_with::{{closure}}
1: std::panicking::rust_panic_with_hook
at library/std/src/panicking.rs:581
2: std::panicking::begin_panic_handler::{{closure}}
at library/std/src/panicking.rs:484
3: std::sys_common::backtrace::__rust_end_short_backtrace
at library/std/src/sys_common/backtrace.rs:153
4: rust_begin_unwind
at library/std/src/panicking.rs:483
5: core::panicking::panic_fmt
at library/core/src/panicking.rs:85
6: core::option::expect_none_failed
at library/core/src/option.rs:1234
7: <ethcore::verification::queue::VerificationQueue as core::ops::drop::Drop>::drop
8: core::ptr::drop_in_place
9: core::ptr::drop_in_place
10: alloc::sync::Arc::drop_slow
11: alloc::sync::Arc::drop_slow
12: alloc::sync::Arc::drop_slow
13: alloc::sync::Arc::drop_slow
14: <hashbrown::raw::RawTable as core::ops::drop::Drop>::drop
15: alloc::sync::Arc::drop_slow
16: core::ptr::drop_in_place
17: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll
18: tokio_threadpool::worker::Worker::run_task
19: tokio_threadpool::worker::Worker::run
20: tokio_reactor::with_default
21: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
22: std::sys_common::backtrace::__rust_begin_short_backtrace
23: core::ops::function::FnOnce::call_once{{vtable.shim}}
24: <alloc::boxed::Box as core::ops::function::FnOnce>::call_once
at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
<alloc::boxed::Box as core::ops::function::FnOnce>::call_once
at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
std::sys::unix::thread::Thread::new::thread_start
at library/std/src/sys/unix/thread.rs:87
25: start_thread
26: __clone

Thread 'event.loop0' panicked at 'Propagating verifier thread panic on shutdown: Any', /openethereum/ethcore/src/verification/queue/mod.rs:853

This is a bug. Please report it at:

https://github.com/openethereum/openethereum/issues/new

Error: 1

[rakita]

There was a problem with rust 1.48, fix is in dev branch, or you can downgrade to 1.47.

[ymonye]

There was a problem with rust 1.48, fix is in dev branch, or you can downgrade to 1.47.

thank you!