Highly available validators #1758
Comments
I am not sure consensus code is the right place to address validator HA aspects, it seems like mixing concerns. Furthermore, we are trying hard to simplify maximally consensus core code as it is critical part of the system, so it can be battle-tested, formally verified and proved correct. I like the idea of logical validator but with different meaning: having multiple validator nodes and multiple signing nodes that act from the outside as a single logical validator that is highly available and never double signs. I was thinking about the following architecture (apologise in advance if something is not fully correct, I am not familiar with KMS details and with SocketPV part of Tendermint). Assume set of Tendermint validator nodes that run normal Tendermint code with the constraint that they can't sign protocol messages. This part is delegated to a signing process that communicates with Tendermint over SocketPV network interface. The signing process is deterministic state machine whose responsibility is signing protocol message and ensuring that the validator does not double sign (therefore, remembering last hight/round/step in which it signed message, and messages itself). This part is maybe implemented (or can be implemented with KMS). Every validator runs Tendermint consensus protocol and although it might receive messages out of order and even different set of messages, decision what block to commit in each height is the same at all validators (by Tendermint consensus guarantees), so validator replicas are kept in sync by Tendermint consensus protocol. The temporary local differences at validator nodes are not important as the signing process will sign only single message (for all validators replica) for each height/round/step. So set of validators replica plus the replicated signing process look from external world as a single deterministic process. Implementing replicated signing process requires relying on state machine library for crash tolerant model (Raft, or Paxos based). The state machine logic shouldn't be very complex as the signing process responsibility is relatively simple (signing messages and remembering what has been signed). I am not sure if it exists good open sourced RSM (Raft or Paxos based) library out there. |
As it has been the basis for a wide variety of infrastructure projects over the last years there is a plethora of tested implementations. for Raft. Paxos itself is less popular as far as implementations go, but there CASPaxos which is a very recent attempt to iterate on the shortcomings of its predecessors. |
I think this is what we want to go for, and I think we can do it all through the SocketPV (ie. without changes to Tendermint) It should be noted that whenever you replicate key material across multiple HSMs you change the security model, even if you use a consensus algorithm across them, unless you build the consensus rules into the HSM itself. Otherwise there's no way for an HSM to know for certain "there was no signature produced for height X". In any case, I don't think this is something that should be built into Tendermint any more than it already is (ie. the SocketPV). Really what we're talking about is a new program between Tendermint and KMS that implements the SocketPV but restricts forwarding messages to the actual KMS based on some consensus protocol. |
Here's one way to think about how to build that intermediate layer. We could actually do it as an ABCI app running on its own Tendermint cluster - let's call it So we have the validator process, say Suppose we also give each clusign node an ID so we can distinguish who should sign. The process could be like this:
Note a few things about this solution:
|
Note there is also the possibility of implementing a BLS pubkey type for validator signing. This would allow for multiple KMS to service a single validator, but it doesn't solve the problem of validator replicas. Also, AFAIKVZ (as far as I know via zaki ...), there's no constant time impls of BLS yet |
The above was mostly for illustration purposes around how to do this with a consensus algorithm and Tendermint. We can work towards that. A simpler short term solution would be to build the
I would encourage folks to try to build something like this, but to do it as a layer between Tendermint and the KMS. |
@ebuchman I conceptually like the idea of "Tendermint all the way down". If I understand the model correctly, it should protect against both gaiad and KMS becoming SPOF. For this to work, clusign-tm should create a block as soon as it sees a tx - but that shouldn't be a problem. I agree with @milosevic that Tendermint and consensus code should be kept as simple as possible. At the same time we need to explore how to provide production level HA at validator level, in a way that will make it "fairly easy" and cost effective for validators to setup (so they will actually use it) For the purpose of network liveness and deterministic block times, from my perspective validator HA and consensus are closely related. Conceptually I don't see it as a mixing of concerns. At the end of the day, any solution that solves SPOF of both gaiad and KMS with minimal changes gets my |
I am also all for "Tendermint all the way down". We will probably make it possible in the future to have |
I like proposed solution, but if there is a much simpler algorithm out there (without BFT and leader rotation, leader based, no logs even better), folks may be better with it (given there is a {their language} battle-tested implementation). But again, I'd encourage people to experiment and figure out the best possible solution. |
I think this is out of scope for this repo. We have a couple of OSS solutions already and some closed source ones as well. |
I am aware of the closed source ones, but AFAIK there is not any open source products? Would be fairly cool to see the "tendermint all the way down" implemented... But yes, this might not be the right repo. |
I think relying double-signing risk onto some third party made kms solution is not a practical solution because the risk we are talking about is meaning almost a death sentence of the business for a validator. I will never use that third party made kms even if it is made by most trusted developers. The trustness we need here is way higher than trustness provided by any kms. Then, alternative solution is, to build kms by each validator. It also seems not very practical since most validator does not have enough technical skills to trust their own-made kms. So, I think the ultimate solution to this is for tendermint to allow a validator using multiple consensus key without causing double-signing. In this way, validators can safely setup active-passive or active-active setup without creating a new failure point, KMS. Because double-signing is the most important risk of each validator, the system(tendermint) should care about the solution for the problem. And I think KMS is not a general solution for most validators. Below is our suggestion about multiple consensus key rotation. multiple consensus key rotation
positive effect
negative effect
I created new issue(#4076) to continue the discussion. |
As per tendermint/tmkms#29, I am not convinced that adding in redundancy at the KMS level is the right place to do it.
Assuming the redundant KMS runs on hardware isolated from the validator (gaiad), the validator itself would still remaing a single point of failure.
An idea I have been floating earlier is that of a "logical validator" concept, which I believe can also be made safe in the context of double signing (though I am not a consensus wizard). If done right, it should AFAICS be highly available and allow for simple and cheap deployment.
A logical validator means a number of gaiad instances signing with the same public key (via KMS or with locally stored key, doesn't matter). Let's imagine each such instance runs on its own dedicated hardware and call it a physical validator.
Each physical validator is assigned a priority, which is must be unique among all the physical validators that constitutes the logical validator.
All physical validators are active at the same time and behaves exactly as they do today. The only difference is that they include their unique priority when voting (the priority is signed as well).
Simply put, the idea is that the existing consensus code is modified (slightly) to ensure a logical validator is always participating in consensus as long as a one of the physical validators that constitute is, is alive.
Physical validators might observe transactions in a different order, essentially double signing. This is resolved at the consensus layer, by only considering input of the physical validator that has the highest priority and ignoring the rest of the votes from the same public key (logical validator).
To put this in active/passive terms: the physical validator with the highest priority, that manages to vote in a round, is thus considered the "active" part while the remaining physical validators (sharing same pubkey) are considered "passive". The existing consensus code thus takes care of failing over and there are no new dependencies introduced.
Double signing (with same priority and public key) should of course still result in slashing, but this should no longer by a risk of failing over.
Depending on the implementation details at the consensus layer, it seems to me that this might provide good availability while introducing little complexity.
It also lowers the barrier to entry to setup a HA validator, which should translate to more determistic block times overall.
Also see Virtual Router Redundancy Protocol.
Referencing #870.
The text was updated successfully, but these errors were encountered: