no space left for the block in the unincluded segment #3282

seunlanlege · 2024-02-09T18:04:38Z

Is there an existing issue?

I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

This is not a support question.

Description of bug

Hyperbridge collator on rococo is panicking with this message.

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
      std::panicking::rust_panic_with_hook
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:649:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
   4: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   5: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   6: cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::maybe_drop_included_ancestors
   7: frame_support::storage::transactional::with_transaction
   8: <cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}
   9: environmental::using_once
  10: <gargantua_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter
  11: <gargantua_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch
  12: <sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply
  13: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic
  14: gargantua_runtime::api::dispatch
  15: environmental::using
  16: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
  17: sc_executor::wasm_runtime::RuntimeCache::with_instance
  18: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  19: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
  20: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  21: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  22: <gargantua_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_consensus_aura::AuraApi<__SrApiBlock__,sp_consensus_aura::sr25519::app_sr25519::Public>>::__runtime_api_internal_call_api_at
  23: sp_block_builder::BlockBuilder::apply_extrinsic
  24: <gargantua_runtime::RuntimeApiImpl<Block,C> as sp_api::ApiExt<Block>>::execute_in_transaction
  25: sc_basic_authorship::basic_authorship::Proposer<Block,C,A,PR>::propose_with::{{closure}}
  26: <sc_basic_authorship::basic_authorship::Proposer<Block,C,A,PR> as sp_consensus::Proposer<Block>>::propose::{{closure}}
  27: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  28: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  29: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  30: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  31: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  32: tokio::runtime::task::core::Core<T,S>::poll
  33: tokio::runtime::task::harness::Harness<T,S>::poll
  34: tokio::runtime::blocking::pool::Inner::run
  35: std::sys_common::backtrace::__rust_begin_short_backtrace
  36: core::ops::function::FnOnce::call_once{{vtable.shim}}
  37: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys/unix/thread.rs:108:17
  38: <unknown>
  39: <unknown>


Thread 'tokio-runtime-worker' panicked at 'no space left for the block in the unincluded segment', /home/seun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.7.0/src/lib.rs:1339

This is a bug. Please report it at:

	https://github.com/paritytech/cumulus/issues/new



====================

Version: 0.3.0-7c2f45b77f9

   0: sp_panic_handler::set::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
      std::panicking::rust_panic_with_hook
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:657:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
   4: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   5: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   6: core::panicking::panic_display
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:178:5
      core::panicking::panic_str
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:152:5
      core::option::expect_failed
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/option.rs:1985:5
   7: <cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize
   8: <(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize
   9: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook
  10: frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block
  11: gargantua_runtime::api::dispatch
  12: environmental::using
  13: sc_executor::executor::WasmExecutor<H>::with_instance::{{closure}}
  14: sc_executor::wasm_runtime::RuntimeCache::with_instance
  15: <sc_executor::executor::NativeElseWasmExecutor<D> as sp_core::traits::CodeExecutor>::call
  16: sp_state_machine::execution::StateMachine<B,H,Exec>::execute
  17: <sc_service::client::call_executor::LocalCallExecutor<Block,B,E> as sc_client_api::call_executor::CallExecutor<Block>>::contextual_call
  18: <sc_service::client::client::Client<B,E,Block,RA> as sp_api::CallApiAt<Block>>::call_api_at
  19: <gargantua_runtime::RuntimeApiImpl<__SrApiBlock__,RuntimeApiImplCall> as sp_consensus_aura::AuraApi<__SrApiBlock__,sp_consensus_aura::sr25519::app_sr25519::Public>>::__runtime_api_internal_call_api_at
  20: sp_block_builder::BlockBuilder::finalize_block
  21: sc_basic_authorship::basic_authorship::Proposer<Block,C,A,PR>::propose_with::{{closure}}
  22: <sc_basic_authorship::basic_authorship::Proposer<Block,C,A,PR> as sp_consensus::Proposer<Block>>::propose::{{closure}}
  23: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
  24: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
  25: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
  26: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
  27: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  28: tokio::runtime::task::core::Core<T,S>::poll
  29: tokio::runtime::task::harness::Harness<T,S>::poll
  30: tokio::runtime::blocking::pool::Inner::run
  31: std::sys_common::backtrace::__rust_begin_short_backtrace
  32: core::ops::function::FnOnce::call_once{{vtable.shim}}
  33: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys/unix/thread.rs:108:17
  34: <unknown>
  35: <unknown>


Thread 'tokio-runtime-worker' panicked at 'set_validation_data inherent needs to be present in every block!', /home/seun/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cumulus-pallet-parachain-system-0.7.0/src/lib.rs:265

This is a bug. Please report it at:

	https://github.com/paritytech/cumulus/issues/new

Steps to reproduce

No response

The text was updated successfully, but these errors were encountered:

alexggh · 2024-02-12T07:40:23Z

@seunlanlege
Can you give us a bit more details about what collator versions you are running ?

seunlanlege · 2024-02-12T08:24:05Z

@seunlanlege Can you give us a bit more details about what collator versions you are running ?

Hi @alexggh, We are currently on the Polkadot v1.7.0 release from crates.io. Here's my Cargo.toml, We've also disabled async backing in the runtime on rococo and are using the basic aura consensus engine. I've attached logs from the past 24 hrs. We're noticing a lot of re-orgs of parachain blocks.
gargantua.logs.txt

alexggh · 2024-02-12T12:39:05Z

Looked a bit at the code, I can't spot any obvious mistake, I would recommend going again through https://wiki.polkadot.network/docs/maintain-guides-async-backing for the necessary setup.

I've attached logs from the past 24 hrs. We're noticing a lot of re-orgs of parachain blocks.

The logs are empty for me, could you re-upload them.

seunlanlege · 2024-02-12T12:45:36Z

The logs are empty for me, could you re-upload them.

Yeah sorry about that, here they are.

gargantua.logs.txt.zip

I would recommend going again through https://wiki.polkadot.network/docs/maintain-guides-async-backing for the necessary setup.

Yeah already did all of this

alexggh · 2024-02-12T14:39:44Z

Looked a bit more closer, it seems the parachains is progressing, is my understanding correct ?

gargantua.logs 2.txt:482873:2024-02-12 12:41:24.605  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5276 (2 ms) [hash: 0x6775c34796d8651f29782d436ef0884f284525313bab7ede524ded451f56be6d; parent_hash: 0x5f45…9eb2; extrinsics (2): [0x9de9…1fca, 0x9711…77b7]
gargantua.logs 2.txt:482952:2024-02-12 12:41:36.565  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5277 (2 ms) [hash: 0x1f7b1b5aa6d82e878486b221a2c494bd789b23c8d1cdc830a7f70a16238c4f8f; parent_hash: 0xc3ed…0021; extrinsics (2): [0x349a…0e7a, 0xa939…cb7f]
gargantua.logs 2.txt:483029:2024-02-12 12:41:48.478  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5278 (2 ms) [hash: 0x2c3f3eba0a7be93d497ad61f3596b67f11e04cc6fd0711605f83a9cc43a88c06; parent_hash: 0xaee7…dfe9; extrinsics (2): [0xefdc…b938, 0xdb50…ba8c]
gargantua.logs 2.txt:483037:2024-02-12 12:41:48.681  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5278 (2 ms) [hash: 0x921ab1afac39d0987c9f5f8f0e06f902746b72d9a747388486f772e68598c88d; parent_hash: 0xaee7…dfe9; extrinsics (2): [0xefdc…b938, 0x9977…b941]
gargantua.logs 2.txt:483181:2024-02-12 12:42:00.584  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5279 (2 ms) [hash: 0x47a245a81d4b82b7ec0f78bc9f98841e6b040742c84d068fec4ac7ba351b9f7c; parent_hash: 0x2e55…a961; extrinsics (2): [0xcbd8…9c36, 0xeda5…cf6e]
gargantua.logs 2.txt:483259:2024-02-12 12:42:12.582  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5280 (2 ms) [hash: 0x0acb4320fa9cd698d5a2b899598ce503bcd9fc8ca1cee615bea0cf1a04e4be2c; parent_hash: 0xda3e…89db; extrinsics (2): [0x30a2…9a6d, 0xb70c…4a44]
gargantua.logs 2.txt:483336:2024-02-12 12:42:24.488  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: 🎁 Prepared block for proposing at 5281 (2 ms) [hash: 0x1e9aa5b42dc191c121859b200f60ad2545831b6db73c5f2a11d1f1fa2f82dda2; parent_hash: 0x3b9a…dcfd; extrinsics (2): [0x3874…8577, 0xa504…3d49]

Then I see, two types of errors coming from time to time.
1.

WARN tokio-runtime-worker wasm-runtime: Evicting failed runtime instance error=Runtime panicked: no space left for the block in the unincluded segment

and
2.

2024-02-11 20:28:43.231  WARN tokio-runtime-worker wasm-runtime: Evicting failed runtime instance error=Runtime panicked: Slot must increase

Error 2. is harmless and it is caused by the fact that it seems that your collator tries to build blocks every 6seconds and the runtime refuses it, are you running the lookahead collator, if yes, that might be also the reason for 1, but it is hard to tell without more details, so can you also provide:

Logs were at least debug level is enabled for aura.
Parachain id for this, I wasn't able to identify from the logs or in polkadot.js for rococo.

seunlanlege · 2024-02-12T15:49:02Z

Looked a bit more closer, it seems the parachains is progressing, is my understanding correct ?

Yes it's progressing, but frequently gets stalled on a single block and needs to re-org.

Error 2. is harmless and it is caused by the fact that it seems that your collator tries to build blocks every 6seconds and the runtime refuses it, are you running the lookahead collator

We were using the basic collator, as we have a cli flag to select which to use, but i just now switched to the lookahead collator and the runtime errors are gone. Recent logs with aura=trace
gargantua.logs.txt

eskimor · 2024-02-12T16:17:17Z

Superficially it looks like a parameter mismatch. It seems the parachain is producing more blocks ahead of time than it is supposed to.

seunlanlege · 2024-02-12T16:33:18Z

Superficially it looks like a parameter mismatch. It seems the parachain is producing more blocks ahead of time than it is supposed to.

Yeah shouldn't this mean that the basic collators should be deprecated?

s0me0ne-unkn0wn · 2024-02-12T17:32:57Z

I have a feeling that's the same problem we discussed with @bkchr recently and came to the conclusion that those panics are harmless as long as the chain progresses. Reorgs are a different problem (see #3205)

@bkchr WDYT?

Given how the block production is driven for Parachains right now, with the enabling of async backing we would produce two blocks per slot. Until we have a proper collator implementation, the "hack" is to prevent the production of multiple blocks per slot. Closes: #3282

seunlanlege added I10-unconfirmed Issue might be valid, but it's not yet known. I2-bug The node fails to follow expected behavior. labels Feb 9, 2024

alexggh self-assigned this Feb 12, 2024

bkchr mentioned this issue Feb 13, 2024

Parachains-Aura: Only produce once per slot #3308

Merged

bkchr closed this as completed in #3308 Feb 13, 2024

arrudagates mentioned this issue May 9, 2024

Evicting failed runtime instance error=Runtime panicked: Slot must increase InvArch/InvArch-Node#158

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

no space left for the block in the unincluded segment #3282

no space left for the block in the unincluded segment #3282

seunlanlege commented Feb 9, 2024 •

edited

Loading

alexggh commented Feb 12, 2024

seunlanlege commented Feb 12, 2024 •

edited

Loading

alexggh commented Feb 12, 2024 •

edited

Loading

seunlanlege commented Feb 12, 2024

alexggh commented Feb 12, 2024 •

edited

Loading

seunlanlege commented Feb 12, 2024

eskimor commented Feb 12, 2024

seunlanlege commented Feb 12, 2024

s0me0ne-unkn0wn commented Feb 12, 2024

no space left for the block in the unincluded segment #3282

no space left for the block in the unincluded segment #3282

Comments

seunlanlege commented Feb 9, 2024 • edited Loading

Is there an existing issue?

Experiencing problems? Have you tried our Stack Exchange first?

Description of bug

Steps to reproduce

alexggh commented Feb 12, 2024

seunlanlege commented Feb 12, 2024 • edited Loading

alexggh commented Feb 12, 2024 • edited Loading

seunlanlege commented Feb 12, 2024

alexggh commented Feb 12, 2024 • edited Loading

seunlanlege commented Feb 12, 2024

eskimor commented Feb 12, 2024

seunlanlege commented Feb 12, 2024

s0me0ne-unkn0wn commented Feb 12, 2024

seunlanlege commented Feb 9, 2024 •

edited

Loading

seunlanlege commented Feb 12, 2024 •

edited

Loading

alexggh commented Feb 12, 2024 •

edited

Loading

alexggh commented Feb 12, 2024 •

edited

Loading