Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Panic: Trie lookup error: Database missing expected key #1733

Closed
xlc opened this issue Feb 8, 2019 · 17 comments

Comments

5 participants
@xlc
Copy link
Contributor

commented Feb 8, 2019

3 validator nodes, running mostly fine (one of them keep falling behind for some reason). 3 more full nodes, all 3 full nodes crashed after running for a while
restart gives following error

2019-02-07 21:58:52 Roles: FULL
2019-02-07 21:58:53 Best block: #8003
2019-02-07 21:58:53 Local node address is: /ip4/0.0.0.0/tcp/30333/p2p/QmPsnPYNiz3VMpJXvxwGBWcdRVKHKbTvtFnnrxc2NL9Xm6
2019-02-07 21:58:53 Kademlia random query has yielded empty results
2019-02-07 21:58:53 Listening for new connections on 0.0.0.0:9944.
2019-02-07 21:58:53 Idle (0 peers), best: #8003 (0xd0b8…1828), finalized #1593 (0x24e0…8a93), ⬇ 0 ⬆ 0
====================
stack backtrace:
   0: substrate_panic_handler::set::{{closure}}::h8c6bfb324807e0af (0x560aca78d64f)
   1: rust_panic_with_hook (0x560acab6249d)
             at src/libstd/panicking.rs:482
   2: continue_panic_fmt (0x560acab61f51)
             at src/libstd/panicking.rs:385
   3: rust_begin_unwind (0x560acab61e35)
   4: panic_fmt (0x560acab7c86c)
             at src/libcore/panicking.rs:85
   5: core::result::unwrap_failed::h34204b9443806977 (0x560ac9bd7e4d)
   6: <substrate_state_machine::ext::Ext<'a, H, B, T> as substrate_state_machine::Externalities<H>>::storage::h8503c7d781a2f5d4 (0x560ac9c22f2b)
   7: <substrate_executor::wasm_executor::FunctionExecutor<'e, E> as wasmi::host::Externals>::invoke_index::he001b56b6957f29b (0x560ac9d48907)
   8: wasmi::func::FuncInstance::invoke::hc19666a2fa542c8f (0x560ac9ee13c6)
   9: wasmi::runner::Interpreter::start_execution::h27ec5098d427db1a (0x560ac9f429d1)
  10: wasmi::func::FuncInstance::invoke::hc19666a2fa542c8f (0x560ac9ee1452)
  11: wasmi::module::ModuleInstance::invoke_export::he19b32b168d86dfe (0x560aca07c3c8)
  12: substrate_executor::wasm_executor::WasmExecutor::call_in_wasm_module::h04f37870b784637d (0x560ac9d5708b)
  13: <std::thread::local::LocalKey<T>>::with::h1a03f7dfd1bfaa0f (0x560ac9dbfbb7)
  14: substrate_state_machine::execute_using_consensus_failure_handler::h79dbff58735b2520 (0x560ac9ea9594)
  15: <substrate_client::call_executor::LocalCallExecutor<B, E> as substrate_client::call_executor::CallExecutor<Block, substrate_primitives::hasher::blake2::Blake2Hasher>>::call_at_state::h5d81654bfd37cfee (0x560aca016a4b)
  16: <substrate_client::client::Client<B, E, Block, RA>>::lock_import_and_run::{{closure}}::h0284170f83cb78a7 (0x560ac9c1021a)
  17: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::h71ce15e2f718af39 (0x560ac9c008a6)
  18: <substrate_finality_grandpa::import::GrandpaBlockImport<B, E, Block, RA, PRA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::hba40f7ac4511cdeb (0x560ac9dfbed8)
  19: substrate_consensus_common::import_queue::import_many_blocks::h9a4df069f50ed464 (0x560ac9f3d5df)
  20: substrate_consensus_common::import_queue::import_thread::h1aef8acd6e75bb34 (0x560ac9f3be16)
  21: std::sys_common::backtrace::__rust_begin_short_backtrace::h26ef1f51493200e6 (0x560ac9cdc40d)
  22: std::panicking::try::do_call::h2ef0793bbad9113a (0x560ac9edec6f)
  23: __rust_maybe_catch_panic (0x560acab68319)
             at src/libpanic_unwind/lib.rs:92
  24: <F as alloc::boxed::FnBox<A>>::call_box::h6da0ec6574d3aeb7 (0x560aca0c5849)
  25: call_once<(),()> (0x560acab676fd)
             at /rustc/4b1e39b7b36d677803e40130ea29ee6d300abf6e/src/liballoc/boxed.rs:744
      start_thread
             at src/libstd/sys_common/thread.rs:14
      thread_start
             at src/libstd/sys/unix/thread.rs:81
  26: start_thread (0x7f7bb18d6493)
  27: __clone (0x7f7bb1401ace)
  28: <unknown> (0x0)
Thread 'ImportQueue' panicked at 'Externalities not allowed to fail within runtime: "Trie lookup error: Database missing expected key: 0x370bd3c95ab9b31078178fb9c2d6d2806ddf1a9361851ec1190befffdd636422"', src/libcore/result.rs:997

with -l sync=trace

https://gist.github.com/xlc/f79510fe7361090bb752cb5336851eee

@gavofyork

This comment has been minimized.

Copy link
Member

commented Feb 8, 2019

Which commit hash? Substrate or Polkadot?

@gavofyork gavofyork added this to the 1.0 milestone Feb 8, 2019

@xlc

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2019

using this version 11a2418
purge chain fixes it, so must be some race condition

@arkpar

This comment has been minimized.

Copy link
Member

commented Feb 8, 2019

Has this been running an older version with the same database? Could be database corruption caused by one of the issues fixed earlier.

@xlc

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2019

Everything was reset and a new genesis was generated before we upgrade version. It is at block 8000 and we have 6s block time so only been running for ~13h

@xlc

This comment has been minimized.

Copy link
Contributor Author

commented Feb 11, 2019

I think I figured out the cause of this issue, it is relates to the generic asset module we are developing. But I still think it is a substrate bug.

The trouble block contains two transfer of the generic asset.

I had this issue before but a purge chain fixes it so I didn't look into it much. But this time it seems always broken when processing this trouble block.

This is part of the stack trace:

stack backtrace:
   0: substrate_panic_handler::panic_hook::h634713bd606dda3a (0x10a5a1f25)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/panic-handler/src/lib.rs:82
   1: substrate_panic_handler::set::{{closure}}::hd3db95c58b92aaa3 (0x10a5a1b26)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/panic-handler/src/lib.rs:31
   2: std::panicking::rust_panic_with_hook::h13c13167711ed60b (0x10b24f5f1)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/panicking.rs:482
   3: std::panicking::continue_panic_fmt::hb731752a380cf78a (0x10b24f03c)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/panicking.rs:385
   4: _rust_begin_unwind (0x10b24ef28)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/panicking.rs:312
   5: core::panicking::panic_fmt::h33b4e6fb35ab9be8 (0x10b2686b1)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/panicking.rs:85
   6: core::result::unwrap_failed::h1fc4f21bbf65edc3 (0x108d811af)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/macros.rs:16
   7: <core::result::Result<T, E>>::expect::hfd7a0d5c47a9a47f (0x1083de139)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/result.rs:825
   8: <substrate_state_machine::ext::Ext<'a, H, B, T> as substrate_state_machine::Externalities<H>>::storage::{{closure}}::h1d4a18315d58e554 (0x108303543)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/state-machine/src/ext.rs:186
   9: <core::option::Option<T>>::unwrap_or_else::h8679f1bfb086b486 (0x10807b31c)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libcore/option.rs:386
  10: <substrate_state_machine::ext::Ext<'a, H, B, T> as substrate_state_machine::Externalities<H>>::storage::ha819eb2e1f24241e (0x108303313)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/state-machine/src/ext.rs:185
  11: sr_io::read_storage::{{closure}}::h8dca7e65eca9e5fb (0x10a53e27d)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/src/../with_std.rs:64
  12: sr_io::ext::with::{{closure}}::h313b15d6cce34842 (0x10a53e962)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/<::environmental::environmental macros>:24
  13: environmental::with::{{closure}}::h1007dd51f0fd6b28 (0x10a53faa0)
             at /Users/bryanchen/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:105
  14: <std::thread::local::LocalKey<T>>::try_with::ha573de82de8eddff (0x10a53d60a)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/thread/local.rs:299
  15: <std::thread::local::LocalKey<T>>::with::h124fc5d7126353d4 (0x10a53ce5f)
             at /rustc/ceb2512144d1fc26330e85fb9d41c22ba1866259/src/libstd/thread/local.rs:245
  16: environmental::with::h893c2efb66907e22 (0x10a53f96f)
             at /Users/bryanchen/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:99
  17: sr_io::ext::with::ha85b45d26b8280d3 (0x10a53e831)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/<::environmental::environmental macros>:24
  18: sr_io::read_storage::hc0132a9bee046e29 (0x10a53e07c)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/core/sr-io/src/../with_std.rs:64
  19: srml_support::storage::unhashed::get::h84c739c895812c0b (0x108c92377)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/support/src/storage/mod.rs:404
  20: srml_support::double_map::StorageDoubleMap::get::h3f592d82a07da93f (0x108c710cd)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/support/src/double_map.rs:54
  21: <srml_generic_asset::Module<T>>::free_balance::hab4934d2c6498ae1 (0x108c72d88)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/generic-asset/src/lib.rs:186
  22: <srml_generic_asset::Module<T>>::transfer::h8cf71620443f9cc9 (0x108c74380)
             at /Users/bryanchen/.cargo/git/checkouts/substrate-79104a8293a5645e/4e3d744/srml/generic-asset/src/lib.rs:85
  23: <srml_generic_asset::Call<T> as srml_support::dispatch::Dispatchable>::dispatch::hdde289c7fff25f08 (0x108c75ab4)

This is my attempt to translate the stack trace translated to source code
Error starts here: It returns an error if key is missing

let map_e = |e| format!("Trie lookup error: {}", e);

self.backend.storage(key).expect(EXT_NOT_ALLOWED_TO_FAIL))

pub fn read_storage(key: &[u8], value_out: &mut [u8], value_offset: usize) -> Option<usize> {

runtime_io::read_storage(key, &mut [0; 0][..], 0).map(|_| {

unhashed::get(&Self::full_key(k1, k2)[..])

@gavofyork

This comment has been minimized.

Copy link
Member

commented Feb 11, 2019

The database should never miss an expected key. If it's easily reproducible, please provide details - it would make things vastly easier to debug.

@xlc

This comment has been minimized.

Copy link
Contributor Author

commented Feb 11, 2019

I am not sure how should I provide the reproduce steps as it requires connect to my internal testnet which is not public accessible.

I tried purge chain and starts with --pruning=archive which seems fixes this issue so I suspect it relates to db pruning.
I will keep digging it for a while and try to create some minimal reproducible steps.

@arkpar

This comment has been minimized.

Copy link
Member

commented Feb 12, 2019

Logging with -lstate-db=trace would be useful as well.

@xlc

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2019

@arkpar

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

The log looks OK. Could you maybe share the spec file and the archive db somewhere?

@arkpar arkpar added F2-bug 🐜 and removed Z0-unconfirmed labels Feb 15, 2019

@arkpar

This comment has been minimized.

Copy link
Member

commented Feb 15, 2019

Reproduced with the DB provided by @xlc.
Tracing shows that the key was inserted at genesis and removed at block 60. It has not been re-inserted and still somehow got accessed at block 48952. Might be a trie issue.

@rphmeier

This comment has been minimized.

Copy link
Member

commented Feb 18, 2019

maybe trie node encoding is messed up somehow

@arkpar

This comment has been minimized.

Copy link
Member

commented Feb 19, 2019

The issue here is caused by two different branches of the trie having the same leaf node.

Since unlike ethereum we don't hash trie keys, it is possible for the same trie node to be inserted twice into two separate branches of the the same trie. When one branch gets deleted the node is marked as dead, though it is still used in the other branch.

I'll see if I can fix that by changing node hashing scheme to include partial keys.

@JKingdom

This comment has been minimized.

Copy link
Contributor

commented Feb 20, 2019

Hi, we are ChainX team
we meet the same error.
I think it's a critical bug

let me show more detail for this problem
our environment like this:
our project base on the substrate which commit is "e8b558c05ba13a0ee63c7c90e8e4156be38237fd" (2019/2/10)
we run 4 authority node to produce block, and all use --pruning archive mode, thus, it would contain all state.
we run many full node to sync block, and some nodes do not use --pruning archive mode, thus, some node would remove old state.
and when the full node sync blocks,

  • the node which use --pruning archive mode, normally sync block.
  • the node which use default config (no archive, retain 256 block state), all panic at the block number 27043

notice this block height is more than 256

the log is

2019-02-19 15:27:03 Failed to write to trie: Database missing expected key: 0xa7a5129d4b695ce4b72a059eec5bf44f70d5a1109e5e63744597a26f63b1b71a
Hash: given=454954446f6f379655e9eea0a214c16d849c461ee2b89c1742237c00750ac9d4, expected=568678ab87053959fb7da15bfedd6183946085048d0ec1c165b2140672e8e38d
2019-02-19 15:27:04 Syncing  4.6 bps, target=#35560 (1 peers), best: #27023 (0x974a…8580), finalized #27000 (0x026f…1949), ⬇ 0.3kiB/s ⬆ 0

====================

stack backtrace:
   0: substrate_panic_handler::panic_hook::h4c80c61cd518b75a (0x55eec241cd3d)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/panic-handler/src/lib.rs:82
   1: substrate_panic_handler::set::{{closure}}::hbb0ee31e0e60fe13 (0x55eec241c8d6)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/panic-handler/src/lib.rs:31
   2: rust_panic_with_hook (0x55eec27a38bd)
             at src/libstd/panicking.rs:482
   3: std::panicking::begin_panic::ha1e86c9c64bec910 (0x55eec26f67a7)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:412
   4: <chainx_runtime::xexecutive::Executive<System, Block, Context, Payment, Finalisation>>::final_checks::hf22c79d951fbe62c (0x55eec1f2268b)
             at runtime/src/xexecutive.rs:209
   5: <chainx_runtime::xexecutive::Executive<System, Block, Context, Payment, Finalisation>>::execute_block::hd5fd909b1e69198b (0x55eec1f20d9a)
             at runtime/src/xexecutive.rs:105
   6: <chainx_runtime::Runtime as substrate_client::runtime_api::runtime_decl_for_Core::Core<sr_primitives::generic::block::Block<sr_primitives::generic::header::Header<u64, sr_primitives::traits::BlakeTwo256, chainx_runtime::Log>, xr_primitives::generic::unchecked_mortal_compact_extrinsic::UncheckedMortalCompactExtrinsic<<srml_indices::Module<chainx_runtime::Runtime> as sr_primitives::traits::StaticLookup>::Source, u64, chainx_runtime::Call, sr_primitives::Ed25519Signature, u32>>>>::execute_block::hb3260379a48311cc (0x55eec1ed9e33)
             at runtime/src/lib.rs:362
   7: chainx_runtime::api::dispatch::h55b603af0cfa2f54 (0x55eec1f2ccda)
             at runtime/src/lib.rs:351
   8: <chainx_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch::{{closure}}::hddb5cc6fc3329f30 (0x55eec0b43c80)
             at /home/king/workspace/chainpool/ChainX/<::substrate_executor::native_executor::native_executor_instance macros>:14
   9: std::panicking::try::do_call::h3710619e202612ef (0x55eec0b439c3)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:297
  10: __rust_maybe_catch_panic (0x55eec27a9739)
             at src/libpanic_unwind/lib.rs:90
  11: std::panicking::try::h1f32ca7c344d329e (0x55eec0b43852)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:276
  12: std::panic::catch_unwind::h375a35de590fe767 (0x55eec0b43428)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panic.rs:388
  13: substrate_executor::native_executor::safe_call::h74c288476abfb813 (0x55eec0b42a09)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:103
  14: substrate_executor::native_executor::with_native_environment::{{closure}}::h3df228a404f3a843 (0x55eec0b42978)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:112
  15: environmental::using::{{closure}}::h9250eb6207a176df (0x55eec0b43bc1)
             at /home/king/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:90
  16: <std::thread::local::LocalKey<T>>::try_with::h23d632e7182b96e5 (0x55eec0b44139)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:296
  17: <std::thread::local::LocalKey<T>>::with::hef3d00b5cc2bfbd9 (0x55eec0b43f22)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:242
  18: environmental::using::hb0607093eeaaaf53 (0x55eec0b43a76)
             at /home/king/.cargo/registry/src/github.com-1ecc6299db9ec823/environmental-1.0.0/src/lib.rs:67
  19: sr_io::ext::using::h1d464473344411d2 (0x55eec0b4356a)
             at /home/king/workspace/chainpool/ChainX/<::environmental::environmental macros>:21
  20: sr_io::with_externalities::h719101822e3fb9e9 (0x55eec0b4348c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/sr-io/src/../with_std.rs:208
  21: substrate_executor::native_executor::with_native_environment::h0fbf58e04fa0307e (0x55eec0b4292c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:112
  22: <chainx_executor::Executor as substrate_executor::native_executor::NativeExecutionDispatch>::dispatch::h20e69cc2f0c8a80e (0x55eec0b42630)
             at /home/king/workspace/chainpool/ChainX/<::substrate_executor::native_executor::native_executor_instance macros>:13
  23: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::{{closure}}::h8ab54b3d3a93cea7 (0x55eec015f35e)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:261
  24: <std::thread::local::LocalKey<T>>::try_with::h497e4d82a56066f0 (0x55eec0708a6f)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:296
  25: <std::thread::local::LocalKey<T>>::with::h3752aa39db068ae5 (0x55eec06fe032)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/local.rs:242
  26: <substrate_executor::native_executor::NativeExecutor<D> as substrate_state_machine::CodeExecutor<substrate_primitives::hasher::blake2::Blake2Hasher>>::call::ha3517c98fb5047e4 (0x55eec013e1f5)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/executor/src/native_executor.rs:201
  27: substrate_state_machine::execute_using_consensus_failure_handler::h5855500d8941d00a (0x55eec030a62d)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/state-machine/src/lib.rs:338
  28: <substrate_client::call_executor::LocalCallExecutor<B, E> as substrate_client::call_executor::CallExecutor<Block, substrate_primitives::hasher::blake2::Blake2Hasher>>::call_at_state::h2436532f601166fb (0x55eec0448676)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/call_executor.rs:266
  29: <substrate_client::client::Client<B, E, Block, RA>>::block_execution::h9c79ae1700bd1d30 (0x55eebf9b543b)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:806
  30: <substrate_client::client::Client<B, E, Block, RA>>::execute_and_import_block::he1a6f8e4bef63c48 (0x55eebf9c6d1d)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:739
  31: <substrate_client::client::Client<B, E, Block, RA>>::apply_block::he17c4d54d4b2e2c1 (0x55eebf9ace7c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:674
  32: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::{{closure}}::hba9d7af3939c4a98 (0x55eebf9aa76c)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:1301
  33: <substrate_client::client::Client<B, E, Block, RA>>::lock_import_and_run::{{closure}}::hcdc3c6fed815d1c8 (0x55eebf9c1e46)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:608
  34: <substrate_client::client::Client<B, E, Block, RA>>::lock_import_and_run::he5c39686a449a393 (0x55eebf9bc21b)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:621
  35: <substrate_client::client::Client<B, E, Block, RA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::h1d681f15493a7b65 (0x55eebf9aa515)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/client/src/client.rs:1300
  36: <substrate_finality_grandpa::import::GrandpaBlockImport<B, E, Block, RA, PRA> as substrate_consensus_common::block_import::BlockImport<Block>>::import_block::hdd89b149fc03e551 (0x55eec03972e4)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/finality-grandpa/src/import.rs:196
  37: substrate_consensus_common::import_queue::import_single_block::h7890131ffeb8db37 (0x55eec01bdd63)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:459
  38: substrate_consensus_common::import_queue::import_many_blocks::h0518398b3f4a9ded (0x55eec01bb231)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:363
  39: substrate_consensus_common::import_queue::import_thread::h41c99576b7aad8e0 (0x55eec01ba2e0)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:276
  40: <substrate_consensus_common::import_queue::BasicQueue<B, V> as substrate_consensus_common::import_queue::ImportQueue<B>>::start::{{closure}}::hb1817216f0d24873 (0x55eec01b8ced)
             at /home/king/.cargo/git/checkouts/substrate-cce119ab4fc6bea1/5bf3d03/core/consensus/common/src/import_queue.rs:176
  41: std::sys_common::backtrace::__rust_begin_short_backtrace::h9e9cacc9cea5202f (0x55eec04bfbe2)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/sys_common/backtrace.rs:135
  42: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h3aa320fb6f55333d (0x55eec03b8c60)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/mod.rs:469
  43: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hd6ccaeef3b02be4d (0x55eec04e4bf0)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panic.rs:309
  44: std::panicking::try::do_call::h87893e184140a9f2 (0x55eec05319a9)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:297
  45: __rust_maybe_catch_panic (0x55eec27a9739)
             at src/libpanic_unwind/lib.rs:90
  46: std::panicking::try::hcf58bda4b4007f7a (0x55eec052f34f)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panicking.rs:276
  47: std::panic::catch_unwind::ha4d988f020d4ad09 (0x55eec04e54d2)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/panic.rs:388
  48: std::thread::Builder::spawn_unchecked::{{closure}}::he82301628f336f6e (0x55eec03b8a05)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/libstd/thread/mod.rs:468
  49: <F as alloc::boxed::FnBox<A>>::call_box::h9ec9f86765a03527 (0x55eec03b91ca)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/liballoc/boxed.rs:734
  50: call_once<(),()> (0x55eec27a8b1d)
             at /rustc/0b7af2668a80fb2fa720a06ca44aff4dd1e9de38/src/liballoc/boxed.rs:744
      start_thread
             at src/libstd/sys_common/thread.rs:14
      thread_start
             at src/libstd/sys/unix/thread.rs:81
  51: start_thread (0x7f74c0b395a9)
  52: __clone (0x7f74c0656cbe)
  53: <unknown> (0x0)

Thread 'ImportQueue' panicked at 'Storage root must match that calculated.', runtime/src/xexecutive.rs:209

we have tried many times, make sure the problem could reproduced every times.
we think this problem is caused by the pruning.

I'm familiar with the MPT structure, I think this problem causes by refactor the MPT nodes, maybe the pruning mode cause unexpect remove for nodes

@gavofyork @arkpar
we could provide our source code and bootsnode to reproduce this problem, if you need code and our bootsnode, please provide the email, I would send those thing for you in private.

think you very much

@arkpar

This comment has been minimized.

Copy link
Member

commented Feb 21, 2019

We've already identified the problem and working on a fix. Related PR: paritytech/trie#12

@xlc

This comment has been minimized.

Copy link
Contributor Author

commented Mar 5, 2019

Any updates on this? @arkpar

@gavofyork

This comment has been minimized.

Copy link
Member

commented Mar 27, 2019

@arkpar shell we try to get paritytech/trie#12 merged and integrated?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.