Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

broken benchmark for pallet_offences (report_offence_im_online) #10027

Closed
bernardoaraujor opened this issue Oct 13, 2021 · 14 comments · Fixed by #10782
Closed

broken benchmark for pallet_offences (report_offence_im_online) #10027

bernardoaraujor opened this issue Oct 13, 2021 · 14 comments · Fixed by #10782
Assignees
@ggwpez
Labels
I3-bug The node fails to follow expected behavior.
Projects
[OLD] Benchmarking and Weights

Comments

@bernardoaraujor
Copy link
Contributor

I want to run all possible benchmarks on FRAME.

I have substrate checked out at monthly-2021-10. While running the benchmarks on my machine (i7 CPU, 16Gb RAM), I got stuck with the following error:

$ ./target/release/substrate benchmark --chain dev --steps 50 --repeat 20 --pallet=* --extrinsic=*
...
2021-10-13 01:49:58 Running Benchmark:	pallet_offences	report_offence_im_online	214/50	0/1    
2021-10-13 01:50:10 Running Benchmark:	pallet_offences	report_offence_im_online	215/50	0/1    

====================

Version: 3.0.0-dev-bf9683eee4-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:626:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:519:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
   4: rust_begin_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
   5: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
   6: core::panicking::assert_failed_inner
   7: core::panicking::assert_failed
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking>::run_benchmark
  10: <node_runtime::Runtime as frame_benchmarking::utils::runtime_decl_for_Benchmark::Benchmark<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,u32>,node_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<node_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<node_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<node_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<node_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<node_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<node_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<node_runtime::Runtime>)>>>>::dispatch_benchmark
  11: node_runtime::api::dispatch
  12: <node_executor::ExecutorDispatch as sc_executor::native_executor::NativeExecutionDispatch>::dispatch
  13: std::thread::local::LocalKey<T>::with
  14: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  15: sc_executor::wasm_runtime::RuntimeCache::with_instance
  16: <sc_executor::native_executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  17: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux
  18: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute
  19: frame_benchmarking_cli::command::<impl frame_benchmarking_cli::BenchmarkCmd>::run
  20: node_cli::command::run
  21: substrate::main
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
  23: std::rt::lang_start::{{closure}}
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:48
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:20
  25: main
  26: __libc_start_main
  27: _start


Thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `2`,
 right: `5`', /home/bear/develop/parity/substrate/substrate/frame/offences/benchmarking/src/lib.rs:358

This is a bug. Please report it at:

	https://github.com/paritytech/substrate/issues/new

2021-10-13 01:50:11 Evicting failed runtime instance: RuntimePanicked("assertion failed: `(left == right)`\n  left: `2`,\n right: `5`")    
Error: Input("Error executing and verifying runtime benchmark: RuntimePanicked(\"assertion failed: `(left == right)`\\n  left: `2`,\\n right: `5`\")")

Is this an actual bug, or did I do something wrong when I started the benchmarking?
@github-actions github-actions bot added the J2-unconfirmed Issue might be valid, but it’s not yet known. label Oct 13, 2021
@bernardoaraujor
Copy link
Contributor Author

bernardoaraujor commented Oct 14, 2021 •
edited
Loading

same error with monthly-2021-09 and monthly-2021-08

@bernardoaraujor bernardoaraujor changed the title broken benchmark broken benchmark for pallet_offences (report_offence_im_online) Oct 14, 2021
@bernardoaraujor
Copy link
Contributor Author

bernardoaraujor commented Oct 15, 2021 •
edited
Loading

slightly different error message on monthly-2021-07:

$ ./target/release/substrate benchmark --chain dev --wasm-execution=compiled --steps 50 --repeat 20 --pallet=pallet_offences --extrinsic=*
...
2021-10-15 16:48:54 Benchmarking report_offence_im_online 15/17, run 6/20    
2021-10-15 16:48:59 Benchmarking report_offence_im_online 16/17, run 5/20    

====================

Version: 2.0.0-83808aa815-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:626:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:519:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
   4: rust_begin_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
   5: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
   6: core::panicking::assert_failed_inner
   7: core::panicking::assert_failed
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults>>::run_benchmark::{{closure}}
  10: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults>>::run_benchmark
  11: <node_runtime::Runtime as frame_benchmarking::utils::runtime_decl_for_Benchmark::Benchmark<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,u32>,node_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<node_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<node_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<node_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<node_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<node_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<node_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<node_runtime::Runtime>)>>>>::dispatch_benchmark
  12: node_runtime::api::dispatch
  13: std::thread::local::LocalKey<T>::with
  14: <node_executor::Executor as sc_executor::native_executor::NativeExecutionDispatch>::dispatch
  15: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  16: sc_executor::wasm_runtime::RuntimeCache::with_instance
  17: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  18: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux
  19: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute
  20: frame_benchmarking_cli::command::<impl frame_benchmarking_cli::BenchmarkCmd>::run
  21: node_cli::command::run
  22: substrate::main
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
  24: std::rt::lang_start::{{closure}}
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:48
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:20
  26: main
  27: __libc_start_main
  28: _start


Thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `4`', /home/bear/develop/parity/benchmark-report/substrate/frame/offences/benchmarking/src/lib.rs:336

This is a bug. Please report it at:

	https://github.com/paritytech/substrate/issues/new

Error: Input("Error executing runtime benchmark: RuntimePanicked(\"assertion failed: `(left == right)`\\n  left: `1`,\\n right: `4`\")")

I was forced to add --wasm-execution here, otherwise I would run into the following:

error: 'Interpreted' isn't a valid value for '--wasm-execution <METHOD>'
	[possible values: compiled, interpreted-i-know-what-i-do]

@ferrell-code
Copy link
Contributor

ferrell-code commented Oct 19, 2021 •
edited
Loading

I believe you need to use the flag --features runtime-benchmarks to benchmark pallets :)

@bernardoaraujor
Copy link
Contributor Author 

$ git show -s
commit bf9683eee40f82cc4e01a05cd375b0e8bba3c8ef (HEAD, tag: monthly-2021-10, tag: devhub/latest)
Author: Shawn Tabrizi <shawntabrizi@gmail.com>
Date:   Thu Sep 30 12:46:48 2021 -0400

    Simple Trait to Inspect Metadata (#9893)
    
    * simple trait to inspect metadata
    
    * import vec
    
    
$ ./target/release/substrate --version
substrate 3.0.0-dev-bf9683eee4-x86_64-linux-gnu


$ ./target/release/substrate benchmark --chain dev --steps 50 --repeat 20 --pallet=* --extrinsic=* --features runtime-benchmarks
error: Found argument '--features' which wasn't expected, or isn't valid in this context

USAGE:
    substrate benchmark --chain <CHAIN_SPEC> --extrinsic <extrinsic> --pallet <pallet> --repeat <repeat> --steps <steps>

For more information try --help

@ferrell-code
Copy link
Contributor

ferrell-code commented Oct 19, 2021 •
edited
Loading

Oh yeah the feature flag is for cargo to compile the substrate binary not a flag for substrate, see example command below

cargo run --features runtime-benchmarks -- benchmark --chain dev --wasm-execution=compiled --steps 50 --repeat 20 --pallet=pallet_offences --extrinsic=*

@kianenigma
Copy link
Contributor

slightly different error message on monthly-2021-07:

$ ./target/release/substrate benchmark --chain dev --wasm-execution=compiled --steps 50 --repeat 20 --pallet=pallet_offences --extrinsic=*
...
2021-10-15 16:48:54 Benchmarking report_offence_im_online 15/17, run 6/20    
2021-10-15 16:48:59 Benchmarking report_offence_im_online 16/17, run 5/20    

====================

Version: 2.0.0-83808aa815-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:626:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:519:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
   4: rust_begin_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
   5: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
   6: core::panicking::assert_failed_inner
   7: core::panicking::assert_failed
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults>>::run_benchmark::{{closure}}
  10: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults>>::run_benchmark
  11: <node_runtime::Runtime as frame_benchmarking::utils::runtime_decl_for_Benchmark::Benchmark<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,u32>,node_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<node_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<node_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<node_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<node_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<node_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<node_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<node_runtime::Runtime>)>>>>::dispatch_benchmark
  12: node_runtime::api::dispatch
  13: std::thread::local::LocalKey<T>::with
  14: <node_executor::Executor as sc_executor::native_executor::NativeExecutionDispatch>::dispatch
  15: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  16: sc_executor::wasm_runtime::RuntimeCache::with_instance
  17: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  18: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux
  19: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute
  20: frame_benchmarking_cli::command::<impl frame_benchmarking_cli::BenchmarkCmd>::run
  21: node_cli::command::run
  22: substrate::main
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
  24: std::rt::lang_start::{{closure}}
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:48
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:20
  26: main
  27: __libc_start_main
  28: _start


Thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `4`', /home/bear/develop/parity/benchmark-report/substrate/frame/offences/benchmarking/src/lib.rs:336

This is a bug. Please report it at:

	https://github.com/paritytech/substrate/issues/new

Error: Input("Error executing runtime benchmark: RuntimePanicked(\"assertion failed: `(left == right)`\\n  left: `1`,\\n right: `4`\")")

I was forced to add --wasm-execution here, otherwise I would run into the following:

error: 'Interpreted' isn't a valid value for '--wasm-execution <METHOD>'
	[possible values: compiled, interpreted-i-know-what-i-do]

The last note clearly shows that your code is outdated. The default execution has been compiled for almost 4 months now:

substrate/utils/frame/benchmarking-cli/src/lib.rs

Line 131 in bd2e5f3

default_value = "compiled"

@bernardoaraujor
Copy link
Contributor Author

slightly different error message on monthly-2021-07:

$ ./target/release/substrate benchmark --chain dev --wasm-execution=compiled --steps 50 --repeat 20 --pallet=pallet_offences --extrinsic=*
...
2021-10-15 16:48:54 Benchmarking report_offence_im_online 15/17, run 6/20    
2021-10-15 16:48:59 Benchmarking report_offence_im_online 16/17, run 5/20    

====================

Version: 2.0.0-83808aa815-x86_64-linux-gnu

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:626:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:519:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/sys_common/backtrace.rs:141:18
   4: rust_begin_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:515:5
   5: core::panicking::panic_fmt
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/panicking.rs:92:14
   6: core::panicking::assert_failed_inner
   7: core::panicking::assert_failed
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults>>::run_benchmark::{{closure}}
  10: <pallet_offences_benchmarking::Pallet<T> as frame_benchmarking::utils::Benchmarking<frame_benchmarking::utils::BenchmarkResults>>::run_benchmark
  11: <node_runtime::Runtime as frame_benchmarking::utils::runtime_decl_for_Benchmark::Benchmark<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,u32>,node_runtime::Call,sp_runtime::MultiSignature,(frame_system::extensions::check_spec_version::CheckSpecVersion<node_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<node_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<node_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<node_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<node_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<node_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<node_runtime::Runtime>)>>>>::dispatch_benchmark
  12: node_runtime::api::dispatch
  13: std::thread::local::LocalKey<T>::with
  14: <node_executor::Executor as sc_executor::native_executor::NativeExecutionDispatch>::dispatch
  15: sc_executor::native_executor::WasmExecutor::with_instance::{{closure}}
  16: sc_executor::wasm_runtime::RuntimeCache::with_instance
  17: <sc_executor::native_executor::NativeExecutor<D> as sp_core::traits::CodeExecutor>::call
  18: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute_aux
  19: sp_state_machine::execution::StateMachine<B,H,N,Exec>::execute
  20: frame_benchmarking_cli::command::<impl frame_benchmarking_cli::BenchmarkCmd>::run
  21: node_cli::command::run
  22: substrate::main
  23: std::sys_common::backtrace::__rust_begin_short_backtrace
  24: std::rt::lang_start::{{closure}}
  25: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/core/src/ops/function.rs:259:13
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:48
      std::panicking::try::do_call
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:401:40
      std::panicking::try
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panicking.rs:365:19
      std::panic::catch_unwind
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/panic.rs:434:14
      std::rt::lang_start_internal
             at /rustc/c8dfcfe046a7680554bf4eb612bad840e7631c4b/library/std/src/rt.rs:45:20
  26: main
  27: __libc_start_main
  28: _start


Thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `1`,
 right: `4`', /home/bear/develop/parity/benchmark-report/substrate/frame/offences/benchmarking/src/lib.rs:336

This is a bug. Please report it at:

	https://github.com/paritytech/substrate/issues/new

Error: Input("Error executing runtime benchmark: RuntimePanicked(\"assertion failed: `(left == right)`\\n  left: `1`,\\n right: `4`\")")

I was forced to add --wasm-execution here, otherwise I would run into the following:

error: 'Interpreted' isn't a valid value for '--wasm-execution <METHOD>'
	[possible values: compiled, interpreted-i-know-what-i-do]

The last note clearly shows that your code is outdated. The default execution has been compiled for almost 4 months now:

substrate/utils/frame/benchmarking-cli/src/lib.rs

Line 131 in bd2e5f3

default_value = "compiled"

yeah I'm aware.
I kept rolling back the tags and running the benchmarks again, to see whether the error message would go away.
When I hit monthly-2021-07 the execution flag behavior changed.

But that's somewhat secondary to this issue. My original goal was just to report this error message I found when benchmarking pallet_offences.

@ferrell-code
Copy link
Contributor

ferrell-code commented Oct 26, 2021 •
edited
Loading

yeah my advice was useless 💀 apologies.

substrate/frame/offences/benchmarking/src/lib.rs

Line 408 in 732f037

assert_eq!(
that is the assert that is failing in the benchmark and the source of the panic, idk if it really matters though

@thiolliere
Copy link
Contributor

maybe the benchmark expect some specific configuration and the way you configured the runtime is inconsistent with this expectation.
So either the configuration of runtime is not sane and the expectation of the benchmark are good or the benchmark is not generic enough on the configuration and needs to be fixed.

@kianenigma
Copy link
Contributor

I get the same issue on current master. It needs a fix. Probably a configuration is broken in node-runtime (and is seemingly fine in polkadot-runtime because we run those benchmarks there and they are all fine? )

@kianenigma kianenigma added I3-bug The node fails to follow expected behavior. and removed J2-unconfirmed Issue might be valid, but it’s not yet known. labels Nov 12, 2021
@kianenigma kianenigma added this to Backlog in Runtime via automation Nov 12, 2021
@thiolliere
Copy link
Contributor

sidenote: out of curiosity where is the result of those benchmark used ?

@shawntabrizi
Copy link
Member

@ggwpez

@ggwpez ggwpez mentioned this issue Jan 18, 2022
Merged
16 tasks
@ggwpez
Copy link
Member

ggwpez commented Jan 20, 2022 •
edited
Loading

I will look into this now.

sidenote: out of curiosity where is the result of those benchmark used ?

Babe and Grandpas benchmarks should depend on it, but currently don't. (They need fixing AFAIK)
PS: paritytech/polkadot-sdk#395
@thiolliere

@ggwpez ggwpez self-assigned this Jan 20, 2022
@ggwpez
Copy link
Member

ggwpez commented Jan 24, 2022

Somehow the one test that works, only checks the event emission when run in test config.

substrate/frame/offences/benchmarking/src/lib.rs

Line 365 in 542a858

#[cfg(test)]

If I add this to the failing event tests in report_offence_grandpa, then it also works.
Does anyone know, why it is only checking correct event count for test config?

@ggwpez ggwpez added this to To do in [OLD] Benchmarking and Weights via automation Feb 2, 2022
This was referenced Aug 24, 2023
Open
Merged
Runtime automation moved this from Backlog to Done Feb 4, 2022
[OLD] Benchmarking and Weights automation moved this from To do to Done Feb 4, 2022
@shawntabrizi shawntabrizi moved this from Done to Archive in Runtime Mar 3, 2022
@shawntabrizi shawntabrizi moved this from Done to Archive in [OLD] Benchmarking and Weights Mar 3, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@ggwpez ggwpez

Labels
I3-bug The node fails to follow expected behavior.
Projects
Benchmarking and Weights
Status: Archive
[OLD] Benchmarking and Weights
  
Archive
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Make offences benchmark runnable paritytech/substrate
6 participants
@shawntabrizi @kianenigma @bernardoaraujor @ggwpez @thiolliere @ferrell-code