How to persist an actor?

[RussellLuo]

While maintaining lots of actors in a cluster, chances are that we need to gracefully restart a node for version update (similar to this orleans issue).

To gracefully restart a node, one possible solution is persisting all actors on that node before shutdown, and then recovering all the persisted actors after the node is started. And for persisting an actor, we need to persist both the actor's internal state and all the pending messages in the actor's mailbox.

The current persistence package seems to be dedicated to this need:

// Embed Mixin to gain the capability to store events and snapshots.
type Actor struct {
    persistence.Mixin
    // other fields
}

func (a *Actor) Receive(ctx actor.Context) {
    switch msg := ctx.Message().(type) {
    case *MyMessage:
        a.PersistReceive(msg)
        // other operations
    case *persistence.RequestSnapshot:
        a.PersistSnapshot(&MySnapshot{})
    case *MySnapshot:
        // recover internal state from snapshot
    case *persistence.ReplayComplete:
        // all events (received after the recovered snapshot) are replayed
    }
}

As shown above, the persisting operations (i.e. PersistReceive() and PersistSnapshot()), by design, can only be triggered from within an actor, which means we can only persist the messages received by the actor. But as far as I can see, it's now impossible to get messages in the mailbox from within an actor. So how can we persist the pending messages, which has not been received by the actor?

Any thoughts on this? Or is there any plan to enhance the persistence package to add support for persisting all pending messages of an actor?