facilitate timeout faults

[ggutoski]

As discussed, we have two tasks to facilitate timeout fault processing:

• Do not proceed until next round until all parties have received all the messages they expect to receive. I think this is pretty simple. All that's needed is to modify expecting_more_msgs_this_round so that (i) it is the same for all parties, and (ii) it returns false only when we have received messages from all parties. Example: Currently, Alice might proceed to the next round even if she has not yet seen a p2p message from Bob to Charlie. Instead, Alice should not proceed until she has received all messages, including those from Bob to Charlie.
• Add a new API call waiting_on that returns a list (Vec<Vec<Crime>>) of shares that have not yet sent the messages we expect them to send. All parties should deduce the same list. Axelar-core will use this list to vote on who has timed out.

Assigning to @sdaveas but I can advise on the change to expecting_more_msgs_this_round.

[sdaveas]

As discussed, we can respond to the Abort message with the list of parties we have not received an answer from.
Note that malicious parties can choose not to respond to Abort message. Need to think if this complicates our task.

[sdaveas]

As discussed, there is the following problem:

Let A, B and C. Each needs to send bcasts. When R1 starts, each party has already self-delivered his bcast to himself, so their states is
A: {A, _, _ }, B:{ _, B, _ }, C:{ _, _, C}.
all parties agree that they are expecting more messages

A sends bcast:
A: {A, _, _ }, B:{ A, B, _ }, C:{ A, _, C}.
all parties agree that they are expecting more messages

B sends bcast:
A: {A, B, _}, B:{ A, B, _ }, C:{ A, B, C}.
A and B agree that they expect more messages but C is ready to proceed to R1

One way to avoid that is to remove self delivery from tofn and rely on the broker (axelar-core) to send our message back to us.

[sdaveas]

What is the best fit for crimes associated with timeouts?

Stall{victim: usize, msg_type: MsgType} is the most detailed one, because it reviles which message was skipped towards which party (and of course the criminal from the index of the crime inside the crime list).
When we have a broadcast message though, this victim field is redundant because all parties are victims. There are the following options:

1. wrap it inside an Option
2. set it to usize:: MAX
3. remove victim altogether. It doesn't have practical usage as for the rest of the behaviours

[sdaveas]

Actually we should do opt 3 because all p2p message types already include a to field which is always the victim in our case.

[ggutoski]

Actually we should do opt 3 because all p2p message types already include a to field which is always the victim in our case.

Agreed