How to allow actors to export metrics when egress policy is enforced?

[haiyanmeng]

Assumption: metrics collection relies on gRPC h2c.

Short-term solution: Add the metrics collection address into the egress policy of every actor needing to export metrics. This can be tedious. A default egress policy in ActorTemplate #1324 should help some. But if a Substrate cluster uses many different ActorTemplates, the metrics collection address needs to be added into all of them.

Long-term solution: Introduce a defaulting mechanism, which automatically adds the metrics collection address into the egress policy of every actor. This makes the assumption that every actor needs to export metrics, which may not hold.

cc Bowei Du (@bowei) Lior Lieberman (@LiorLieberman) Da Huang (@git286) Eitan Yarmush (@EItanya) Keith Mattix II (@keithmattix)

[bowei]

This issue is framed in terms of a specific stats collector example but I think the user journey describe is more generic. I suggest changing the description and title of the issue and describing the Otel as a specific example of a use case.

[keithmattix]

+1 to Bowei: what's the more fundamental use-case we're solving for here? If the actor needs to make a request to something, it needs an egress policy. If there are infrastructure layer things that we feel like should be transparent, then maybe ateom should instrument it (e.g. maybe ateom scrapes and exports for the actor).

[EItanya]

+1 to Bowei and Keith. I think the other side of this is how to make this sort of default policy easier.

Maybe related is Keith's doc about policy compisition: https://docs.google.com/document/d/1s2klDhbF2mB5sslwUyZBdyXD7JqRt8FWONwJ7wgsomE/edit?tab=t.kzdlrjvbwil#heading=h.pz98xwccc0me

Or my issue about actor template policies #1324

[haiyanmeng]

This issue is framed in terms of a specific stats collector example but I think the user journey describe is more generic. I suggest changing the description and title of the issue and describing the Otel as a specific example of a use case.

Good point. Renamed the title and updated the description.

[git286]

For Actor Telemetry, the long term recommended solution is that substrate should relay the telemetry to the collector instead of letting actor directly pushing data to collector through network. See #853 for the long term solution proposal.

The ask of egress policy for collector is a short term solution when the network lockdown is enforced but telemetry relay has not implemented.

Agreed the general problem of making default egress policy easier still exists when it's needed.

[haiyanmeng]

For short-term, we will advise users to add the metrics collection address into the egress policy of every actor needing to export metrics. After a default egress policy in ActorTemplate #1324 is available, we will advise users to add the metrics collection address into the egress policy of an ActorTemplate to avoid adding the address into every actor's egress policy manually.

Da Huang (@git286) , Keith Mattix II (@keithmattix) , Eitan Yarmush (@EItanya) , Bowei Du (@bowei) , what do you think?