Khala blockchain stop producing block due to wasm panic

[h4x3rotab]

Direct error:

[Parachain] panicked at /root/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/72c4535/substrate/frame/aura/src/lib.rs:139:21:
Slot must increase

Situation:

• Khala stopped at [block], 2024-05-01 UTC 06:35
  • Not actually stopped. It turns out the collators were producing blocks but with the panic error.
  • We thought it was stopped because the monitoring system reaches to a RPC node and the RPC node was stucked
• The involved code hasn't been changed for 2+ years
• Neither Khala or Kusama has runtime upgrade
• Last Khala runtime upgrade at [block], [time]
• Last Kusama runtime upgrade at [block], [time]

Follow up:

• After restarting the collators, the frequency of the panic error reduced
• Some internal RPC node was stuck (still checking the reason)

Full panic log:

2024-05-01 07:34:54 [Parachain] panicked at /root/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/72c4535/substrate/frame/aura/src/lib.rs:139:21:
Slot must increase    
2024-05-01 07:34:54 [Parachain] err=Error { inner: Proposing

Caused by:
    0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0xc2f6 - <unknown>!rust_begin_unwind
           1: 0x2da9 - <unknown>!core::panicking::panic_fmt::h3e1dd3d08288569e
           2: 0x40b6dc - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::hb3a22f08cb827fcf
           3: 0x3fecef - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h33aaa37cdfdb2b9c
           4: 0x428a2c - <unknown>!Core_initialize_block
    1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0xc2f6 - <unknown>!rust_begin_unwind
           1: 0x2da9 - <unknown>!core::panicking::panic_fmt::h3e1dd3d08288569e
           2: 0x40b6dc - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55,TupleElement56,TupleElement57,TupleElement58,TupleElement59) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::hb3a22f08cb827fcf
           3: 0x3fecef - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h33aaa37cdfdb2b9c
           4: 0x428a2c - <unknown>!Core_initialize_block }