Discussion: instrumenting servant-client for OpenTelemetry

[lf-]

Hi! I'm a contributor to @iand675's hs-opentelemetry project, which does OpenTelemetry for Haskell.

If you're not familiar, OpenTelemetry lets people trace requests through distributed systems, with any number of services in them, which is very well aligned with use cases that Servant enables. It allows making spans for things that are happening at some point in time, as well as attaching metadata to them. I've used this technology at work to figure out why things are slow, for extracting database statements, and more.

For the servant-server side, OTel instrumentation is easy enough since you can instrument a Servant app at the WAI level, but servant-client is currently not obvious how to instrument.

There are two main things that I would like in a client implementation:

• Propagation of the trace state via headers (so that the recipient server's part of the handling will land in the same trace)
• Instrumentation of how long the request took and what URL it was for

The reason this is hard without something in servant-client is that the current way that hs-opentelemetry-instrumentation-http-client works is that it provides instrumentation-wrapped versions of http-client functions. From what I can tell, there's not a simple way to integrate that into servant-client without applying changes to it, since it directly calls http-client functions.

It's possible that a Servant user could instrument requests by hacking up the HTTP manager by abusing the wrap exception functionality, but that would not account for the time for the server to send the response body.

Thus I'm asking here for ideas and feedback on how to best achieve this.

It would be possible to directly instrument servant-client at the source at very little cost to non-OpenTelemetry users using hs-opentelemetry-api. I'm not super convinced about that idea since there is no canonical OpenTelemetry library for Haskell since opentelemetry also exists (for context, as I understand it, hs-opentelemetry is built more strongly around the OTel library API spec compared to opentelemetry).

[lf-]

Looking into this further, it looks like we could build a hs-opentelemetry-instrumentation-servant-client that implements RunClient, then transform the client monad.

[arianvp]

servant-client seems to be too high level to hook this in in my opinion.

In Golang world otel hooks into the http client by defining the http.RoundTripper interface.
https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http#Transport.RoundTrip

http-client has managerModifyRequest and managerModifyResponse . Would these be enough to inject spans into the requests? They are not exactly the same interface as RoundTripper unfortunately. Wondering if RoundTripper should be added to http-client upstream?

[lf-]

That's been what we've been thinking after some further thoughts on this in the other thread on http-client as well: add some new hooks to Manager to do better instrumentation on that side. It seems like there's support for such a change to http-client so it mostly needs implementing.

Those existing hooks are not enough to do it because they are not tied to the request lifecycle in the right way iirc.

[shinzui]

@lf- Do you know if there were any progress on this in http-client?

[lf-]

Not to my knowledge. I don't personally have the resources to do it and I don't work in haskell full time anymore.

[lf-]

It seems like I did a bad job of remembering to link the issue on that side. It's this one: snoyberg/http-client#494

[shinzui]

It seems like I did a bad job of remembering to link the issue on that side. It's this one: snoyberg/http-client#494

Thank you.