Identify dependencies in the UI when remote is not instrumented

[ndrwrbgs]

Requirement - what kind of business use case are you trying to solve?

Visual distinction of service calls in a trace, when the target service does not implement tracing.

Problem - what in Jaeger blocks you from solving the requirement?

From what I can find in documentation/issues, you can get the serviceName set only if you use the Zipkin ingestion or if there is a span.kind=server span that is receiving your remote call. In my scenario, I'd rather use Jaeger-first APIs if possible, and there is no instrumentation on the receiving side. But it seems that span.kind=client doesn't use the peer.service tag (instead it tries to just hide if there's instrumentation on both ends).

Proposal - what do you suggest to solve the problem or improve the existing situation?

Since the instrumentation-on-both-ends scenario hides the client span anyway, I propose (if there's not a workaround) to use peer.service as the span's service name when there is no server receiving span.

I'm open to any other way to visually distinguish my off-service calls too! (Aside from the workaround that involves spinning up a new Tracer object for each, which was noted in gitter as a workaround but discouraged of its use.

Any open questions to address

[yurishkuro]

When you say "dependencies" are you referring to the service dependency graph? How are you generating it?

[ndrwrbgs]

No, dependencies of my service. I call out to an external service just like the "golden" case, but the difference is the external service isn't instrumented.

My target page is the Trace view.

[yurishkuro]

The Timeline view needs a time-bound span to show a service, which is missing in your case. What kind of display are you suggesting?

[ndrwrbgs]

I think you maybe missed some of my scenario above, but I'm struggling to see what I could improve. Could you help guide me with questions to draw clarification?

For the immediate questions - I have time-bounded spans. I have everything you likely have in some service you trace at work, except take one of those dependencies you call and pretend it can't talk to Jaeger?

[ndrwrbgs]

I tried to take a picture, but of course, my internet isn't playing nicely now ;)

Concretes:
I have code which calls, say, GMail.
Right now in the Jaeger UI it shows MyService::SendEmail. I would prefer if it showed GMail::SendEmail and colored it separately.

I believe from other issues that the UI handles the case peer.service replaces serviceName when span.kind = client and there is a matching one of span.kind = server, and I know that it merges those two spans in some way. Since I don't own GMail, I can't emit the span.kind = server, so I have the client call (that, in a fully instrumented ecosystem would be merged/hidden), and it attributes the call to MyService.

[CodeBlanch]

I opened a similar issue yesterday, probably in the wrong spot: jaegertracing/jaeger#2584

Agree with OP. Right now it really only works if all services in an environment are instrumented and reporting to the same Jaeger. Of course, the real-world use cases are a lot more complicated. Lots of legacy systems out there, third parties, and something like SQL which is unlikely to ever report spans somewhere.

It would be great if Jaeger UI treated span.kind = client/producer + peer.service as the service name and detected/colored spans as a dependency even if no corresponding server span is found. This would be similar to Zipkin RemoteEndpoint behavior, which works great for the same purpose.

Right now you can make it work by breaking up the spans emitted from a process into batches per-service. But that kind of undoes the whole point of the Process part of the API, IMO. It's also way harder on the client to manage all the data into groups in order to send the correct batches. Example of the complexity: https://github.com/open-telemetry/opentelemetry-dotnet/blob/5fa2f105e7fb1e2545ebeb915d6d5e68ba480054/src/OpenTelemetry.Exporter.Jaeger/JaegerExporter.cs#L148-L184

[ovidiubuligan]

This is a sample 3 service demo that even when the remote is instrumented it is still the case (AspnetCore instrumentation and HttpClient instrumentation) :
https://github.com/ovidiubuligan/AspnetOpenTelemetrySample

[yurishkuro]

I have a concrete proposal how this can be implemented visually, using already existing functionality. But default, Jaeger shows client-server calls like this:

The middle span is the client span, which would become the leaf if the route service was not instrumented. When this span is collapsed, the view changes to show the destination service:

I think the same display can be re-purposed for cases when route is not instrumented, but still known via peer.service tag on the client span.

[jpkrohling]

I think the same display can be re-purposed for cases when route is not instrumented, but still known via peer.service tag on the client span.

Interesting idea. This would only touch the jaeger-ui, right?

[yurishkuro]

This would only touch the jaeger-ui, right?

Yes.