Telemetry: publish events for each actor create / restart / terminate

[Aaronontheweb]

Discussed in #5587

^{Originally posted by Aaronontheweb February 7, 2022}
Is your feature request related to a problem? Please describe.

I've been thinking about this in a few areas:

1. Phobos - right now we use invocation counters to keep track of actor starts / stops to keep track the churn of actors. In platforms like Prometheus we compute the delta between start / stop events to compute the total number of alive actors. We can try using things like an OTel UpDownCounter (not yet implemented in .NET) when that gets added to reach a total number and maybe that will be more accurate.
2. Petabridge.Cmd - similar to how we report on shard counts right now, I'd love to update the actor commands to show a breakdown of how many actors are alive by hierarchy position and type.
3. Homespun telemetry solutions - right now most of these are done via the logging system or subscribing to various data channels inside Akka.NET (i.e. Cluster.Subscribe), but the actual structure of the actor hierarchy itself is difficult to capture here.

In particular, some of the work I've been doing around Phobos 2.0 has been leading me to wonder if it makes sense to emit some basic telemetry around actor creation inside Akka.NET itself.

Describe the solution you'd like
I'll have to flesh this out in some more detail, but basically add three EventStream events that can be subscribed to:

public readonly struct ActorStarted{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
}

public readonly struct ActorRestarted{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
   public Exception CrashReason {get;}
}

public readonly struct ActorTerminated{
   public IActorRef Ref {get;} // can be used for topological sort / correlation
   public Type ActorType {get;}
   public TerminatedReason Reason {get;} // stopped, parent decided, system shutting down
}

Having a TerminatedReason would be really helpful - implementing that alone would be a bit of work, but I think it's achievable.

[Aaronontheweb]

closed via #6294