tmkmsv0.4.0 errored out after I stopped gaiad service and will not restart

[phonikg]

I was successfully running tmkms 0.4.0 built with ledgertm support. When I stopped the gaiad service that tmkms was connected to locally I received an error that I was unable to recover from. Here is the error WITH the backtrace results:

BACKTRACE=1

[admin@localhost .tmkms]$ tmkms start
14:23:31 [INFO] tmkms 0.4.0 starting up...
14:23:31 [INFO] [keyring:ledgertm:ledgertm] added validator key cosmosvalconspub1zcjduepq9t8j7hrvyjn7l6sw4ndhlmjt6xwy7dweqsmnxntd5ezkzsccm85q2ctjy6
thread '' panicked at 'index 33 out of range for slice of length 32', libcore/slice/mod.rs:2334:5
note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: std::sys_common::backtrace::print
at libstd/sys_common/backtrace.rs:71
at libstd/sys_common/backtrace.rs:59
2: std::panicking::default_hook::{{closure}}
at libstd/panicking.rs:211
3: std::panicking::default_hook
at libstd/panicking.rs:227
4: std::panicking::rust_panic_with_hook
at libstd/panicking.rs:476
5: std::panicking::continue_panic_fmt
at libstd/panicking.rs:390
6: rust_begin_unwind
at libstd/panicking.rs:325
7: core::panicking::panic_fmt
at libcore/panicking.rs:77
8: core::slice::slice_index_len_fail
at libcore/slice/mod.rs:2334
9: <subtle_encoding::base64::Base64 as subtle_encoding::encoding::Encoding>::decode_to_slice
10: signatory::encoding::decode::Decode::decode_from_file
11: tmkms::client::client_loop

BACKTRACE=full

[admin@localhost .tmkms]$ tmkms start
14:28:30 [INFO] tmkms 0.4.0 starting up...
14:28:30 [INFO] [keyring:ledgertm:ledgertm] added validator key cosmosvalconspub1zcjduepq9t8j7hrvyjn7l6sw4ndhlmjt6xwy7dweqsmnxntd5ezkzsccm85q2ctjy6
thread '' panicked at 'index 33 out of range for slice of length 32', libcore/slice/mod.rs:2334:5
stack backtrace:
0: 0x557791162aef - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hab66bb8ff7088f29
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: 0x55779116c327 - std::sys_common::backtrace::print::h56719ed580ab1bba
at libstd/sys_common/backtrace.rs:71
at libstd/sys_common/backtrace.rs:59
2: 0x557791166ebf - std::panicking::default_hook::{{closure}}::hf8d811a2c97e4b3a
at libstd/panicking.rs:211
3: 0x557791166c24 - std::panicking::default_hook::hc6e051251a5e6b4e
at libstd/panicking.rs:227
4: 0x55779116759e - std::panicking::rust_panic_with_hook::h71214e7ce0f7ac01
at libstd/panicking.rs:476
5: 0x557791167141 - std::panicking::continue_panic_fmt::ha8b8442f4ea9bcac
at libstd/panicking.rs:390
6: 0x557791167025 - rust_begin_unwind
at libstd/panicking.rs:325
7: 0x5577911ac71c - core::panicking::panic_fmt::h0c93626b89c38af6
at libcore/panicking.rs:77
8: 0x5577911a2c71 - core::slice::slice_index_len_fail::h2baf8f016a3596d7
at libcore/slice/mod.rs:2334
9: 0x557791110a47 - <subtle_encoding::base64::Base64 as subtle_encoding::encoding::Encoding>::decode_to_slice::hd99ba65579e92c44
10: 0x5577910ab715 - signatory::encoding::decode::Decode::decode_from_file::h7b3165beafd1a4f9
11: 0x5577910a7f6b - tmkms::client::client_loop::hb49ea8f02e0ca628
12: 0x55779108b647 - std::sys_common::backtrace::__rust_begin_short_backtrace::h58f56d954fc6f171
13: 0x557791094ed1 - std::panicking::try::do_call::h0f7a63942c8a1b30
14: 0x557791177559 - __rust_maybe_catch_panic
at libpanic_unwind/lib.rs:102
15: 0x5577910898f2 - <F as alloc::boxed::FnBox>::call_box::h7a0f8974d1998818
16: 0x557791164cbd - std::sys_common::thread::start_thread::h5213f803a61d7811
at liballoc/boxed.rs:682
at libstd/sys_common/thread.rs:24
17: 0x557791157f55 - std::sys::unix::thread::Thread::new::thread_start::he89121f566d2a8c7
at libstd/sys/unix/thread.rs:90
18: 0x7fcc31b4bdd4 - start_thread
19: 0x7fcc32269eac - __clone
20: 0x0 -

[tarcieri]

If you can reproduce this, a backtrace from debug build would be very helpful (compile without the --release flag).

It looks like it's crashing inside of subtle-encoding while trying to decode your secret_connection.key, ostensibly around here https://github.com/tendermint/kms/blob/master/src/client.rs#L156

[phonikg]

So I ran:
cargo install tmkms --features=ledgertm --force

Then did a export BACKTRACE=full of the newly compiled tmkms to similar result:

[admin@localhost .tmkms]$ tmkms start
15:49:41 [INFO] tmkms 0.4.0 starting up...
15:49:41 [INFO] [keyring:ledgertm:ledgertm] added validator key cosmosvalconspub1zcjduepq9t8j7hrvyjn7l6sw4ndhlmjt6xwy7dweqsmnxntd5ezkzsccm85q2ctjy6
thread '' panicked at 'index 33 out of range for slice of length 32', libcore/slice/mod.rs:2334:5
stack backtrace:
0: 0x55ade4c6baef - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hab66bb8ff7088f29
at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
1: 0x55ade4c75327 - std::sys_common::backtrace::print::h56719ed580ab1bba
at libstd/sys_common/backtrace.rs:71
at libstd/sys_common/backtrace.rs:59
2: 0x55ade4c6febf - std::panicking::default_hook::{{closure}}::hf8d811a2c97e4b3a
at libstd/panicking.rs:211
3: 0x55ade4c6fc24 - std::panicking::default_hook::hc6e051251a5e6b4e
at libstd/panicking.rs:227
4: 0x55ade4c7059e - std::panicking::rust_panic_with_hook::h71214e7ce0f7ac01
at libstd/panicking.rs:476
5: 0x55ade4c70141 - std::panicking::continue_panic_fmt::ha8b8442f4ea9bcac
at libstd/panicking.rs:390
6: 0x55ade4c70025 - rust_begin_unwind
at libstd/panicking.rs:325
7: 0x55ade4cb571c - core::panicking::panic_fmt::h0c93626b89c38af6
at libcore/panicking.rs:77
8: 0x55ade4cabc71 - core::slice::slice_index_len_fail::h2baf8f016a3596d7
at libcore/slice/mod.rs:2334
9: 0x55ade4c19a47 - <subtle_encoding::base64::Base64 as subtle_encoding::encoding::Encoding>::decode_to_slice::hd99ba65579e92c44
10: 0x55ade4bb4715 - signatory::encoding::decode::Decode::decode_from_file::h7b3165beafd1a4f9
11: 0x55ade4bb0f6b - tmkms::client::client_loop::hb49ea8f02e0ca628
12: 0x55ade4b94647 - std::sys_common::backtrace::__rust_begin_short_backtrace::h58f56d954fc6f171
13: 0x55ade4b9ded1 - std::panicking::try::do_call::h0f7a63942c8a1b30
14: 0x55ade4c80559 - __rust_maybe_catch_panic
at libpanic_unwind/lib.rs:102
15: 0x55ade4b928f2 - <F as alloc::boxed::FnBox>::call_box::h7a0f8974d1998818
16: 0x55ade4c6dcbd - std::sys_common::thread::start_thread::h5213f803a61d7811
at liballoc/boxed.rs:682
at libstd/sys_common/thread.rs:24
17: 0x55ade4c60f55 - std::sys::unix::thread::Thread::new::thread_start::he89121f566d2a8c7
at libstd/sys/unix/thread.rs:90
18: 0x7f1a0ac5bdd4 - start_thread
19: 0x7f1a0b379eac - __clone
20: 0x0 -

[tarcieri]

You'll need to do cargo install tmkms --features=ledgertm --force --debug to get a debug build. It defaults to release.

[phonikg]

Ok, so I could recover from it by re generating the secret_connection.key ...Seems key file may just be corrupt?

I backed up the old key and confirmed it will fail if I have it restored.

***Note: If there is anything still to be gained from a debug build let me know and I can run it.

[tarcieri]

Yes, there's definitely a bug in the subtle-encoding Base64 parser.

If the key is no longer valuable, you can send it to me and I can debug from there. Otherwise it seems like I should fuzz that code.

[phonikg]

Hey @tarcieri here is the bad secret file. It is no longer valuable.
secret_connection.zip

[tarcieri]

It looks like this was caused by the handling (or lack thereof) of trailing whitespace in the Base64 decoder (the KMS uses a constant-time Base64 decoder for this as it's decoding a private key).

PR to fix it in an upstream crate here:

https://github.com/iqlusioninc/crates/pull/163/files

[tarcieri]

This should at least error out informing you of the trailing whitespace as of this release: iqlusioninc/crates#164

I will eventually make the parser whitespace tolerant, but for now there are any number of ways to strip trailing whitespace from the key yourself.