Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nightly failure: sanity/rpc_query.py #2544

Closed
SkidanovAlex opened this issue Apr 28, 2020 · 1 comment
Closed

Nightly failure: sanity/rpc_query.py #2544

SkidanovAlex opened this issue Apr 28, 2020 · 1 comment
Assignees
@Kouprin @mikhailOK
Labels
A-chain Area: Chain, client & related

Comments

@SkidanovAlex
Copy link
Collaborator

Fails ocasionally. Example runs:

http://nightly.neartest.com/run/f625606891dec008a572ca61ec37da500c2a95cd_200428_135923
http://nightly.neartest.com/run/2cef4105736ab969470df2751e78fd3f0fc0a81d_200428_072213

Failure:

Traceback (most recent call last):
  File "tests/sanity/rpc_query.py", line 38, in <module>
    query_result1 = fix_result(nodes[-2].get_account("test%s" % i))
  File "tests/sanity/rpc_query.py", line 35, in fix_result
    result["result"]["block_hash"] = None
KeyError: 'result'

In this run:
http://nightly.neartest.com/run/b693daeca9a871c0fd96cf9960551a047597c403_200427_140713

It caused the third node to crash:

thread 'main' panicked at 'RuntimeError::StorageInconsistentState, StorageInconsistentState("Trie node missing")', neard/src/runtime.rs:1046:17
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1069
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1427
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: <alloc::boxed::Box<F> as core::ops::function::Fn<A>>::call
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/liballoc/boxed.rs:1031
  11: near_primitives::test_utils::init_stop_on_panic::{{closure}}::{{closure}}
             at core/primitives/src/test_utils.rs:83
  12: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:515
  13: rust_begin_unwind
             at src/libstd/panicking.rs:419
  14: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:373
  15: <neard::runtime::NightshadeRuntime as near_chain::types::RuntimeAdapter>::obtain_state_part
             at neard/src/runtime.rs:1046
  16: near_chain::chain::Chain::get_state_response_part
             at chain/chain/src/chain.rs:1455
  17: <near_client::view_client::ViewClientActor as actix::handler::Handler<near_network::types::NetworkViewClientMessages>>::handle
             at chain/client/src/view_client.rs:718
  18: <actix::address::envelope::SyncEnvelopeProxy<A,M> as actix::address::envelope::EnvelopeProxy>::handle
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.9.0/src/address/envelope.rs:112
  19: <actix::address::envelope::Envelope<A> as actix::address::envelope::EnvelopeProxy>::handle
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.9.0/src/address/envelope.rs:71
  20: actix::mailbox::Mailbox<A>::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.9.0/src/mailbox.rs:101
  21: <actix::contextimpl::ContextFut<A,C> as core::future::future::Future>::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-0.9.0/src/contextimpl.rs:373
  22: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libcore/future/future.rs:118
  23: tokio::task::core::Core<T>::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/core.rs:128
  24: tokio::task::harness::Harness<T,S>::poll::{{closure}}::{{closure}}
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/harness.rs:120
  25: core::ops::function::FnOnce::call_once
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libcore/ops/function.rs:232
  26: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/panic.rs:318
  27: std::panicking::try::do_call
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/panicking.rs:331
  28: __rust_try
  29: std::panicking::try
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/panicking.rs:274
  30: std::panic::catch_unwind
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/panic.rs:394
  31: tokio::task::harness::Harness<T,S>::poll::{{closure}}
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/harness.rs:101
  32: tokio::loom::std::causal_cell::CausalCell<T>::with_mut
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/loom/std/causal_cell.rs:41
  33: tokio::task::harness::Harness<T,S>::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/harness.rs:100
  34: tokio::task::raw::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/raw.rs:162
  35: tokio::task::raw::RawTask::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/raw.rs:113
  36: tokio::task::Task<S>::run
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/mod.rs:371
  37: tokio::task::local::Scheduler::tick
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:513
  38: <tokio::task::local::LocalFuture<F> as core::future::future::Future>::poll::{{closure}}
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:407
  39: tokio::task::local::Scheduler::with::{{closure}}
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:478
  40: std::thread::local::LocalKey<T>::try_with
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/thread/local.rs:262
  41: std::thread::local::LocalKey<T>::with
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/thread/local.rs:239
  42: tokio::task::local::Scheduler::with
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:474
  43: <tokio::task::local::LocalFuture<F> as core::future::future::Future>::poll
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:402
  44: std::future::poll_with_tls_context
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/future.rs:99
  45: tokio::task::local::LocalSet::run_until::{{closure}}
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:358
  46: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/future.rs:44
  47: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/runtime/basic_scheduler.rs:138
  48: tokio::runtime::Runtime::block_on::{{closure}}
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/runtime/mod.rs:411
  49: tokio::runtime::context::enter
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/runtime/context.rs:72
  50: tokio::runtime::handle::Handle::enter
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/runtime/handle.rs:33
  51: tokio::runtime::Runtime::block_on
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/runtime/mod.rs:408
  52: tokio::task::local::LocalSet::block_on
             at /home/bo_nearprotocol_com/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.9/src/task/local.rs:321
  53: actix_rt::runtime::Runtime::block_on
             at /home/bo_nearprotocol_com/.cargo/git/checkouts/actix-net-8b378701d4b3767e/602db17/actix-rt/src/runtime.rs:89
  54: actix_rt::builder::SystemRunner::run
             at /home/bo_nearprotocol_com/.cargo/git/checkouts/actix-net-8b378701d4b3767e/602db17/actix-rt/src/builder.rs:164
  55: neard::main
             at neard/src/main.rs:199
  56: std::rt::lang_start::{{closure}}
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/rt.rs:67
  57: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  58: std::panicking::try::do_call
             at src/libstd/panicking.rs:331
  59: std::panicking::try
             at src/libstd/panicking.rs:274
  60: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  61: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  62: std::rt::lang_start
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/rt.rs:67
  63: main
  64: __libc_start_main
  65: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
�[2mApr 27 15:14:35.376�[0m �[31mERROR�[0m network: Received error sending message to view client: Mailbox has closed for Some(ed25519:FXXrTXiKWpXj1R6r5fBvMLpstd8gPyrBq3qMByqKVzKF@127.0.0.1:24578)
�[2mApr 27 15:14:35.378�[0m �[34mDEBUG�[0m network: ed25519:Eo9W44tRMwcYcoua11yM7Xfr1DjgR4EWQFM3RU27MEX8: Peer Some(ed25519:He7QeRuwizNEhBioYG3u4DZ8jWXyETiyNzFD3MkTjDMf@127.0.0.1:24577) disconnected.
�[2mApr 27 15:14:35.378�[0m �[34mDEBUG�[0m network: ed25519:Eo9W44tRMwcYcoua11yM7Xfr1DjgR4EWQFM3RU27MEX8: Peer Some(ed25519:5BVKQ5zZKtEUM3AHgj6KAerTpuh3CVZHtAbmVfxLbLhc@127.0.0.1:24580) disconnected.
�[2mApr 27 15:14:35.387�[0m �[34mDEBUG�[0m network: ed25519:Eo9W44tRMwcYcoua11yM7Xfr1DjgR4EWQFM3RU27MEX8: Peer Some(ed25519:FXXrTXiKWpXj1R6r5fBvMLpstd8gPyrBq3qMByqKVzKF@127.0.0.1:24578) disconnected.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "Non-zero exit code: 1" }', neard/src/main.rs:199:13
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.44/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1069
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1427
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: <alloc::boxed::Box<F> as core::ops::function::Fn<A>>::call
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/liballoc/boxed.rs:1031
  11: near_primitives::test_utils::init_stop_on_panic::{{closure}}::{{closure}}
             at core/primitives/src/test_utils.rs:83
  12: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:515
  13: rust_begin_unwind
             at src/libstd/panicking.rs:419
  14: core::panicking::panic_fmt
             at src/libcore/panicking.rs:111
  15: core::option::expect_none_failed
             at src/libcore/option.rs:1211
  16: core::result::Result<T,E>::unwrap
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libcore/result.rs:1003
  17: neard::main
             at neard/src/main.rs:199
  18: std::rt::lang_start::{{closure}}
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/rt.rs:67
  19: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  20: std::panicking::try::do_call
             at src/libstd/panicking.rs:331
  21: std::panicking::try
             at src/libstd/panicking.rs:274
  22: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  23: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  24: std::rt::lang_start
             at /rustc/f509b26a7730d721ef87423a72b3fdf8724b4afa/src/libstd/rt.rs:67
  25: main
  26: __libc_start_main
  27: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
@SkidanovAlex SkidanovAlex added A-chain Area: Chain, client & related nightly labels Apr 28, 2020
@SkidanovAlex SkidanovAlex added C-epic Category: an epic and removed C-epic Category: an epic labels Apr 28, 2020
@bowenwang1996
Copy link
Collaborator

The actual failure is StorageInconsistentState: TrieNodeMissing, which is described in this issue #2500

This was referenced Apr 28, 2020
Closed
Closed
@bowenwang1996 bowenwang1996 mentioned this issue May 2, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Kouprin Kouprin

@mikhailOK mikhailOK

Labels
A-chain Area: Chain, client & related
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@SkidanovAlex @Kouprin @mikhailOK @bowenwang1996