Custom Tracing & Metrics

[fledgling-vish]

Hi Team,

Right now it's not possible to add custom traces & metrics in the restate server using restate API. I think it will be really helpful if we can support this feature. It will provide a streamlined way to do observability.
Some of the use cases it will support:

• Trace third party API calls from the restate server.
• Add information for debugging
• Use custom metrics to trigger alerts through prometheus

[tillrohrmann]

Thanks for creating this issue @fledgling-vish. I've got a couple of clarifying questions to better understand the requirements:

• Do you want to be able to add custom tracing & metrics to the Restate server or a service endpoint (I assume that third API calls would originate from the user-provided service endpoint)?
• What are concrete metrics you would like to expose?
• What type of information would you like to see to better debug Restate?

[fledgling-vish]

@tillrohrmann I was talking about adding custom tracing & metrics to restate server ( not the restate cluster). May be the right solve for this is to just add otel libraries to the restate server itself.

From the restate-cluster, it might be a good idea to expose some service level metrics as opposed to just bucketing everything into ingress metrics. for example, if a service fail with a Terminal Error, it might be interesting to have it captured in a metric etc.

[jackkleeman]

, it might be a good idea to expose some service level metrics as opposed to just bucketing everything into ingress metrics. for example, if a service fail with a Terminal Error, it might be interesting to have it captured in a metric etc.

We definitely intend to add this!

May be the right solve for this is to just add otel libraries to the restate server itself.

I think so, yes, at least for now

[slinkydeveloper]

I think a low hanging fruit that can unlock bunch of the points discussed here is to make sure we pass through the otel context propagated from the invoker: restatedev/sdk-java#76

The invoker right now is already passing a w3c trace context header to the sdk, we just need to make sure that the user can in their code access (using the otel APIs) this context and attach additional data to it. For example, in Java this roughly translates to adding the trace context to the current otel context. This is probably similar in the TS sdk.

I'll open a similar issue in TS SDK, thanks for reporting @fledgling-vish

[fledgling-vish]

Thanks @slinkydeveloper let me know if i can help with it.