RFC 104: Internal messaging using the actor model

[thanethomson]

📖 Rendered

I'm submitting this as an RFC as part of a broader effort to consider different options in simplifying Comet's internal concurrency model (which is unnecessarily complex right now). The actor model is only one possible option, and I thought it would be worth capturing some of my past explorations on employing the model somewhere.

Again, this is an RFC and not an ADR because it's merely exploring the solution space as opposed to proposing a concrete course of action. The conclusion of the RFC is that adopting an actor framework is not desirable for a whole host of reasons, and neither is adopting a full-blown actor model. We can, however, extract some principles or practices from the model that could be beneficial.

---

PR checklist

• Tests written/updated
• Changelog entry added in .changelog (we use unclog to manage our changelog)
• Updated relevant documentation (docs/ or spec/) and code comments

[mzabaluev]

I'm wary about converting to a fully type-erased actor model. The disadvantages are as you have outlined, and without exhaustive integration test coverage, there will be a possibility of programmer errors that are easily caught by strict type checking.

I'd suggest a hybrid approach: make each reactor testable in isolation with dependency services provided by handles, but define typed APIs for actor handles that can be used to send messages to a particular type of actor. We shouldn't have too many of them in CometBFT.

For the sake of programmer sanity, it would be good if the request-response model is usable with goroutines. As in, to program an asyncronous flow involving a request roundtrip to another actor, one should not have to pass callbacks, manually rig up response matching at the local actor's message processing loop, or anything of the sort. It should be just sequential code. Unfortunately, this is not Rust and we can't have async/await to make the suspension points explicit.

[thanethomson]

@mzabaluev I totally agree. That for me is one of the biggest reasons not to use the actor model (at least the strict version implemented by most actor frameworks).

That's also why I submitted this as an RFC as opposed to an ADR - it's just capturing some of the tradeoffs of the architectural approach. I, like you, believe that there's definitely a better way, but there are some subtle things we could learn from the actor model that would help address some of the tech debt in the system.

[adizere]

Thanks Thane for documenting this. The conclusion is sound and the discussion is very useful for future reference.

[sergefdrv]

Following up your discussion, I'd like to add my 5 cents. In my opinion, one big disadvantage of the actor model is the lack of built-in flow control (a.k.a. backpressure) between single-threaded actors. What should an actor do if it's not ready to process some incoming message? How to ensure bounded memory usage if individual actors make progress at different pace?

Another concern is fragmentation of the logic in code. Actor's code would need to be implemented as a state machine reacting on incoming events (messages). If certain processes happen to be split across several rounds of communication between actors then the overall logic becomes segregated and harder to follow.

[thanethomson]

Good points @sergefdrv!

In my opinion, one big disadvantage of the actor model is the lack of built-in flow control (a.k.a. backpressure) between single-threaded actors.

Is there some kind of general approach that you'd recommend considering here? Given my understanding of our internal protocols, each one's approach to handling backpressure would possibly be different.

Another concern is fragmentation of the logic in code. Actor's code would need to be implemented as a state machine reacting on incoming events (messages). If certain processes happen to be split across several rounds of communication between actors then the overall logic becomes segregated and harder to follow.

In my mind this is best addressed by modelling protocol state machines separately to the code that mutates them, with state machines being thread-safe but single-/un-threaded (i.e. no goroutines or threads ever spawned from within the state machine).

Did you have any recommendations in mind though on this front as to good patterns to consider?

[sergefdrv]

@thanethomson

Is there some kind of general approach that you'd recommend considering here? Given my understanding of our internal protocols, each one's approach to handling backpressure would possibly be different.

I'd recommend sticking to pull-based mechanisms, which I distinguish from poll- and push-based. In pull-based interaction the consumer connects to the producer and fetches the items provided by the producer if they are ready, or waits until a new item becomes available. As apposed to push-based approaches, the producer cannot force the consumer to have to deal with a new item if the consumer is not ready. Also, in contrast to poll-based approaches, the consumer doesn't have to repetitively check for new items.

I think one can relatively naturally build such mechanism on top of Go channels. A big advantage of using Go channels, comparing e.g. to blocking methods, is the possibility to combine several blocking operations in a single select statement.

... modelling protocol state machines separately to the code that mutates them, with state machines being thread-safe but single-/un-threaded (i.e. no goroutines or threads ever spawned from within the state machine).

I suppose, by mutation, you mean restructuring the state machine. In any case, if we want to implement a distributed atomic broadcast mechanism (sequence consensus) then we have to merge inherently concurrent paths of execution into a single-threaded sequential execution at some point. I suspect the earlier we try to get rid of the concurrency in code the more we have to fragment otherwise cohesive protocol logic. In particular, I find sigle-threaded state machines are often not really straightforward to follow, especially if hand-coded. Async/await certainly helps a lot, but it's a quite low-level feature, which without additional structure easily results in a great mess. Regarding Go, I just came across this article about coroutines in Go, which you might want to look at.

Did you have any recommendations in mind though on this front as to good patterns to consider?

I think I'm yet to discover a decent approach to implementing distributed protocols; this is the whole point of the Replica_IO project, which I'm working on. I have some promising stuff to look at more closely on this list. Luckily, I'm starting from scratch and don't have to bother with any legacy burden at this point.