Panic when opening CFD

[da-kami]

2022-12-06 07:51:53  INFO ten_ten_one::cfd::open: Adding custom output
thread 'frb_workerpool' panicked at 'assertion failed: self.channel_state.try_lock().is_ok()', /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning/src/ln/channelmanager.rs:2820:23
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
flutter: {_CfdOrderConfirmationState} {openCfd} {Failed to open CFD.} {FfiException(PANIC_ERROR, assertion failed: self.channel_state.try_lock().is_ok(), null)} {LogLevel.ERROR} {06 December 2022 06:51:53 PM}

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning/src/ln/channelmanager.rs:5686:68
thread 'frb_workerpool' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning/src/ln/channelmanager.rs:1849:59
[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: FfiException(PANIC_ERROR, called `Result::unwrap()` on an `Err` value: PoisonError { .. }, null)
#0      FlutterRustBridgeBase._transformRust2DartMessage (package:flutter_rust_bridge/src/basic.dart:101:9)
#1      FlutterRustBridgeBase.executeNormal.<anonymous closure> (package:flutter_rust_bridge/src/basic.dart:49:9)
<asynchronous suspension>
#2      ChannelChangeNotifier.init.<anonymous closure> (package:ten_ten_one/wallet/channel_change_notifier.dart:94:28)
<asynchronous suspension>

thread 'frb_workerpool' panicked at 'called `Result::unwrap()` on an `Err` value: PoisonError { .. }', /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning/src/ln/channelmanager.rs:1849:59
thread 'frb_workerpool' panicked at 'called `Result::unwrap()` on an `Err` value: Any { .. }', /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning-background-processor/src/lib.rs:562:43
stack backtrace:
   0:        0x10b8f1222 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd2e8cbde22b780fc
   1:        0x10b90f96a - core::fmt::write::hd6692086cdd356a7
   2:        0x10b8ea50c - std::io::Write::write_fmt::h6043124a2486acbb
   3:        0x10b8f2b7b - std::panicking::default_hook::{{closure}}::h87a12b8b06887dd7
   4:        0x10b8f2887 - std::panicking::default_hook::h7f68dad17e0bfaa4
   5:        0x10b8f319f - std::panicking::rust_panic_with_hook::hd57427cbbfc3717a
   6:        0x10b8f30e3 - std::panicking::begin_panic_handler::{{closure}}::h33aab6d04e2bba70
   7:        0x10b8f16b8 - std::sys_common::backtrace::__rust_end_short_backtrace::h0e7a76f927db9964
   8:        0x10b8f2dad - _rust_begin_unwind
   9:        0x10b95c183 - core::panicking::panic_fmt::hcf6f3c517c6f3cb3
  10:        0x10b95c2e5 - core::result::unwrap_failed::ha988429942445917
  11:        0x10a7ff9f9 - core::result::Result<T,E>::unwrap::hf717198b306f2a37
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/result.rs:1107:23
  12:        0x10a7ffc48 - lightning_background_processor::BackgroundProcessor::join_thread::h76ba56613d3fbf4a
                               at /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning-background-processor/src/lib.rs:562:20
  13:        0x10a7ffba8 - lightning_background_processor::BackgroundProcessor::stop_and_join_thread::hb87ae5b745b5efd6
                               at /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning-background-processor/src/lib.rs:557:3
  14:        0x10a7ffc71 - <lightning_background_processor::BackgroundProcessor as core::ops::drop::Drop>::drop::h6ff72e039f98e26f
                               at /Users/dakami/.cargo/git/checkouts/rust-lightning-42dd281c301507a3/10b69c9/lightning-background-processor/src/lib.rs:570:3
  15:        0x10a801aa5 - core::ptr::drop_in_place<lightning_background_processor::BackgroundProcessor>::h91145a69c29b17b1
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ptr/mod.rs:487:1
  16:        0x109df8ee0 - ten_ten_one::api::run_ldk::{{closure}}::h90f0cf26728f7630
                               at /Users/dakami/CoBloX/repos/10101/rust/src/api.rs:149:9
  17:        0x109e61885 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h4b2756eaa2e7891e
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/future/mod.rs:91:19
  18:        0x10a4fe4a6 - <core::pin::Pin<P> as core::future::future::Future>::poll::h17985a4f9461c974
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/future/future.rs:124:9
  19:        0x109e3f96a - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}::hcee73df61b7a3ee3
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:48
  20:        0x10a5eb1a3 - tokio::coop::with_budget::{{closure}}::h69b8c3afe8d0e666
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9
  21:        0x10a10fb36 - std::thread::local::LocalKey<T>::try_with::he70e4d5e3b69b920
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:445:16
  22:        0x10a108a3c - std::thread::local::LocalKey<T>::with::hea8c087f778b3db4
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/local.rs:421:9
  23:        0x109e3e355 - tokio::coop::with_budget::hccef5655278e900b
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5
  24:        0x109e3e355 - tokio::coop::budget::h096d71ef2217213a
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5
  25:        0x109e3e355 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::h18f683dd59c10703
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:525:25
  26:        0x109e305bc - tokio::runtime::scheduler::current_thread::Context::enter::hd83b94abbe61aad5
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:349:19
  27:        0x109e3daad - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::he304eb406be5414a
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:524:36
  28:        0x109e36732 - tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}::hbc857bb0d06777cb
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:57
  29:        0x10a43cc3b - tokio::macros::scoped_tls::ScopedKey<T>::set::haeef6081e929d75c
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/macros/scoped_tls.rs:61:9
  30:        0x109e35f88 - tokio::runtime::scheduler::current_thread::CoreGuard::enter::hb6a19eb54ac0b7b6
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:595:27
  31:        0x109e36e10 - tokio::runtime::scheduler::current_thread::CoreGuard::block_on::ha2bb2b44d3ecc953
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:515:19
  32:        0x109e2a163 - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::hd710c03da686b3c6
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/current_thread.rs:161:24
  33:        0x10a4803e4 - tokio::runtime::Runtime::block_on::hd68a551eaf94dbf1
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/mod.rs:490:46
  34:        0x10a26e5c1 - ten_ten_one::api::run_ldk::h37813cd51886e4b8
                               at /Users/dakami/CoBloX/repos/10101/rust/src/api.rs:154:5
  35:        0x10a24bdb1 - ten_ten_one::bridge_generated::bridge_generated::wire_run_ldk_impl::{{closure}}::{{closure}}::hcb7c4aa670f7ee80
                               at /Users/dakami/CoBloX/repos/10101/rust/src/bridge_generated/bridge_generated.rs:103:38
  36:        0x109e7fca1 - <flutter_rust_bridge::handler::ThreadPoolExecutor<EH> as flutter_rust_bridge::handler::Executor>::execute::{{closure}}::{{closure}}::he1d8a7609dddfed2
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/flutter_rust_bridge-1.51.0/src/handler.rs:217:27
  37:        0x10a5640ec - std::panicking::try::do_call::h0405f10a1ea4df1b
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492:40
  38:        0x10a56fc9d - ___rust_try
  39:        0x10a55d512 - std::panicking::try::h4d820d8c245ae6cf
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:456:19
  40:        0x10a52d7c8 - std::panic::catch_unwind::h53135a901836a3b3
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panic.rs:137:14
  41:        0x109e79595 - <flutter_rust_bridge::handler::ThreadPoolExecutor<EH> as flutter_rust_bridge::handler::Executor>::execute::{{closure}}::h8b6ea60542d651ab
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/flutter_rust_bridge-1.51.0/src/handler.rs:212:33
  42:        0x109e80e20 - <F as threadpool::FnBox>::call_box::ha66d6c348fd4e527
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/threadpool-1.8.1/src/lib.rs:95:9
  43:        0x10b7bb956 - threadpool::spawn_in_pool::{{closure}}::h3b4e1718559e2487
                               at /Users/dakami/.cargo/registry/src/github.com-1ecc6299db9ec823/threadpool-1.8.1/src/lib.rs:769:17
  44:        0x10b7b2821 - std::sys_common::backtrace::__rust_begin_short_backtrace::hcd428bd14ac87ddc
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys_common/backtrace.rs:122:18
  45:        0x10b7c0271 - std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}::haa9825bdb051c6f3
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/mod.rs:514:17
  46:        0x10b7bcd31 - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::hb753c2036a6b32c6
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/panic/unwind_safe.rs:271:9
  47:        0x10b7afb08 - std::panicking::try::do_call::he54b8adee59119f5
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:492:40
  48:        0x10b7b02cd - ___rust_try
  49:        0x10b7af9ab - std::panicking::try::h5eeebffff4fa9455
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panicking.rs:456:19
  50:        0x10b7bcdd1 - std::panic::catch_unwind::hea2f0eb2a9afd98a
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/panic.rs:137:14
  51:        0x10b7c0111 - std::thread::Builder::spawn_unchecked_::{{closure}}::h7ee84177862b3210
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/thread/mod.rs:513:30
  52:        0x10b7bceb1 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hdc21844abb2f0875
                               at /rustc/897e37553bba8b42751c67658967889d11ecd120/library/core/src/ops/function.rs:248:5
  53:        0x10b8f6607 - std::sys::unix::thread::Thread::new::thread_start::haa45038b11bc331d
  54:     0x7ff813152259 - __pthread_start
thread panicked while panicking. aborting.

[holzeis]

I think I run into the same issue lately. To me, I think this was happening when I tried to open a CFD with a maker margin above the channel funds. Can you reproduce?

[da-kami]

Yes, likely it was above maker funds. We could actually validate that the maker can fulfill the CFD, because we should be able to query the channel amounts and calculate how much the maker needs! That would be a good additional validation step :)

[klochowicz]

regardless of the cause , we could catch the panic and display "failed to open a cfd"

[klochowicz]

I believe we have to find out the real cause of poisoning of this mutex. There was a panic in a call before, and if this happens we can't really do much anymore - some invariants are not held and there's no way we can continue using ldk.
Furthermore, regardless of the cause, we should come up with a way that reinitialises the node after such an error (as it's irrecoverable).

[holzeis]

I guess we could simply call runLdk again in case of an error?

https://github.com/itchysats/10101/blob/6d15d9a42c60e30c8112d449b6fce39306a8e4e3/lib/main.dart#L301-L304