Support for traces across instances

[c23omega]

Hello opentelemetry-ruby team,

we use the gem a microservice platform. There intermediate http requests are made to different services, and therefore we don't have a common webserver instance from which we could create child spans. It seems that to achieve a common span / trace we need to propagate the traceId so we can represent our request structure.

Therefore, first question: Is propagating the traceId a suitable way how to achieve what we need, or are you aware of a better way?

Second, if this is the proper approach, how would this work technically? I've checked the repo, and there i've seen that using custom traceIds is not supported. Therefore i thaught of extending the OpenTelemetry::Trace::Tracer in_span method to accept a trace_id parameter and the add_span method to accept the parameter as well. In the OpenTelemetry::SDK::Trace::Tracer.add_span method the trace_id can then be directly used. Would you consider this as the optimal way how to reach this aim here, or would you suggest a better one? If this is the one we would be happy to contribute with a PR.

Looking forward to your responses,
The best

[ahayworth]

Hi @c23omega - you have the right idea; in order to connect spans from disparate systems you need to propagate a custom trace ID. Luckily, this should be almost - if not completely - free!

I would start by looking at the OpenTelemetry overview document. That page introduces a lot of terminology that's useful when working with the opentelemetry-ruby library. The parts that talk about your question are "Context propagation" - that's the mechanism that carries the trace ID between services on different systems. As long as context is injected into outbound requests and extracted from incoming requests, then spans from different systems should be assembled by your tracing vendor / system automatically.

I would next take a look at our http example - that contains an example of a client and server, and shows you how to manually inject context into the headers of an outgoing request from the client, and how to extract the context from an inbound request on the server. It also shows on the server how you activate the context that was extracted - and that activation is what causes the server's spans to be logical children of the client's spans.

That example is really good to understand what needs to happen - but what's even better is that you don't actually have to do this manually! If you're using something that has auto-instrumentation available, then it's almost zero-effort!

For example, if your "client" application uses faraday to make its requests, then you can initialize the OpenTelemetry SDK on the client like so:

require 'opentelemetry/sdk'
require 'opentelemetry-instrumentation-faraday'
OpenTelemetry::SDK.configure do |c|
  c.use 'OpenTelemetry::Instrumentation::Faraday'
end

...and on the "server", if you're using Rails, you could initialize the OpenTelemetry SDK like so:

require 'opentelemetry/sdk'
require 'opentelemetry-instrumentation-rails'
OpenTelemetry::SDK.configure do |c|
  c.use 'OpenTelemetry::Instrumentation::Rails'
end

...and when you do it like that, any outbound request from the client (via Faraday) will have the current context injected automatically into request headers, and any inbound request into the rails server will extract that context and make it the "current" context automatically from the request headers. In that specific example, you don't have to write any code at all other than just initializing the SDKs - the auto-instrumentation handles it all for you.

And, if you're not using faraday and rails, there is a lot of other auto instrumentation available, and they all have examples.

Let me know if that helps!

[c23omega]

Hello @ahayworth,

thank you very much for your information! This helps, at least to some extend ;-). Before asking we checked your examples to see how it works, and in the scope of what was described it works like charm. However, it seems our case is a bit of a corner case and therefore i'm still not sure about the next steps that are the most reasonable now.

How is the Context extracted out of the rails application? Right now we propagate our request information via the X-Request-ID http header, it seems like that this header is not used by ot. If possible, we would prefer not to touch all of our clients since we have a couple of them, and they do a plenty of requests. Can we tell opentelemetry-ruby shomehow to use the X-Request-ID header for detecting the trace id? Is this part how the requests are done also scope of the OpenTelemetry spec? I checked for it but i didn't see it. If so, could you point me where to find it?

Currently i'm thinking of adding the proper headers how ot detects the trace id via a middleware to the env var so ot-ruby can detect it. Do you think this would be a good solution intermediately, until our clients also support ot?

Thanks in advance for your help & BR
c23omega

[ahayworth]

I think I understand:

• You have a Rails server application (or multiple, doesn't matter), and it extracts the X-Request-ID header from clients that send it. If found, it re-uses this request-id rather than generating a new one.
• You have multiple client applications in your control that generate a request-id and set it in X-Request-ID headers for outbound calls.
• This allows a degree of cross-application correlation in a non-tracing world.
• You do not wish to modify the clients (or potentially, cannot modify them).

Based on that understanding, it sounds like you're asking "Can this X-Request-ID be re-used somehow as the trace ID" ?

Right now I don't think there's a good or easy way to do that. The easiest solution is to enable OpenTelemetry on the clients, so that their outgoing HTTP requests will have the correct tracing headers injected automatically. However, you could also create a custom propagator that extracts the header and transforms it into a trace/span ID somehow; if you loaded that into your server application(s) and configured the SDK to use it, then it could work. Take a look at the ottrace propagator for an example of what an implementation looks like.

On a less-technical note - GitHub propagates a request-id around our internal systems in the same way that you do. We've found that it is a very unreliable way to assemble traces, and have stopped relying on it for trace assembly. We now consider it an "extra" signal that is propagated around our systems that can be used to identify a logical request, but it no longer serves as the "glue" that held together an actual trace. It was not really fit for purpose.

[c23omega]

Again many thanks for your answer, that helped a lot. Put it in a nutshell, my question is how can i use the X-Request-ID as a traceID.

You are totally right in that we are passing around X-Request-IDs to trace the internal request flow. This serves for our purposes right now for identifying our traces. Since we have too many clients, we are not able to change them all. Instead, we plan to change our routing part of the application that redirects all the requests, and capture there the content, because there we can identify the relevant telemetry data.

I've checked your link to the ottrace propagator, it looks like exactly what we need. Also the references that are linked there aroun seems to be very helpful. Many thanks for the hint, with this we should be able to resolve our issue.

[ahayworth]

Sounds great, glad it could help!