Separating dispatcher functionality from the mailbox

[samanbarghi]

At the moment, the dispatcher functionality is not independent of the mailbox. Mailbox is responsible for checking the throughput and yielding, which limits the variety of dispatcher that can be used. Also, some dispatchers might require further access to the processing message loop. e.g. a dispatcher might map an actor to a single goroutine without letting it go, or need to process stuff before the goroutine terminates.

My suggestion is to separate the dispatcher functionality from the mailbox, by introducing new functions under the Dispatcher interface and sprinkle those functions in related places in the mailbox code. You can check my suggested approach here: Huawei-CPLLab@7740008

Also, as @rogeralsing suggested it might be better to put back the index variable in the mailbox and pass it to BeforeProcessingMessage function, to make the dispatcher stateless, and thus avoid creating a dispatcher per actor.

On the same note, it might be even better if we can reuse the interception points in MailboxStatistics to achieve this.

[samanbarghi]

@rogeralsing If we want to use the MailboxStatistics, then we need to also call ms.MessageReceived when a system message is being processed. Otherwise, this can lead to starvation of other actors if an actor receives a sequence of systemMessages. Or MailboxStatistics should be expanded to include ms.SystemMessageReceived so the total number of messages can be deduced from sum of user+sys messages.

[rogeralsing]

Mailbox is responsible for checking the throughput and yielding

You raise a valid concern. we do have the EndpointWriterMailbox in remote which does not even have a throughput setting, it simply reads x messages and send that as a batch to the target actor.
This makes me think that maybe the throughput setting shouldnt even be on the dispatcher?
As different mailbox implementations might have different ways to deal with how it consumes messages.

Maybe the dispatcher should only be responsible for scheduling some task onto whatever thread/goroutine/pool there is?

Thoughts on that?

[samanbarghi]

Let me start by saying that I think the dispatchers in golang cannot be as flexible as how AKKA use them. Go runtime does not provide the same flexibility as AKKA, e.g. there is no way to create an OS thread on demand. But at the same time, it provides nice features that omit the need for certain dispatchers. e.g., in AKKA, I/O heavy actors could be dispatched using a pinnedDispatcher which is used to create an additional java thread to handle that actor to avoid blocking other actors. But Go runtime handles that automatically in case a goroutine is blocked in a syscall for a long time.

So there are few cases that dispatchers can cover, i.e. calling goroutine dispatcher, where the calling actor's goroutine handle the destination's mailbox. Or goroutine pool dispatcher. As far as it goes with interacting with go runtime.

Some of the functionality I added such as AfterStart and BeforeTerminate or BeforeBatchStart are not necessary for the cases that can be implemented. e.g. AfterStart and BeforeTerminate can be added to a dispatcher by creating a wrapper function for processMessage in the dispatcher, pass the wrapper to the goroutine and write the AfterStart and beforeTerminate code before and after calling processMessage. But at the same time, I think dispatcher should have some control over when processMessage returns, e.g. if a goroutine needed to be used to process many mailboxes, the dispatcher should be able to force the mailbox to stop processing messages, even if there are still messages in the mailbox, and continue with another mailbox.

Having said that, I think the whole processMessage functionality and schedulerStatus are features of the dispatcher not the mailbox. The throughput number is also a feature of the dispatcher, along with the counter that keeps track of the number of messages processed. What if, I need a dispatcher that uses time quantum instead of number of messages to address fairness and liveliness? Although current design is good for performance, but I feel does not provide enough flexibility down the road for having variety of dispatchers.

Another issue is what if there is a dispatcher that requires a specific kind of mailbox? how can this dependency be addressed?