go/keymanager/client(?): Intermittent errors when committe size > 1

[Yawning]

I have been told that there is retry logic, and both the rust enclave RPC and go key manager clients appear to have something that looks like retry logic.

That said, when there is more than one node in a key manager committee, a simple non-enclave key manager request (via gRPC) from rust code intermittently fails with the following panic:

Testing key manager connection via gRPC transport...
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: RpcFailure(RpcStatus { status: Unknown, details: Some("worker/keymanager: error from runtime: decrypt error") })', tests/clients/simple-keyvalue-enc/src/main.rs:99:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Node logs: https://buildkite.com/oasislabs/oasis-core-ci/builds/3614 (The keymanager-replication tests in the 6/7 non-sgx group)

I assume something, somewhere should be re-establishing the session as part of the retry logic, or something somewhere shouldn't be as happy to swap to a different node.

[kostko]

It seems this is not yet fixed.