-
Notifications
You must be signed in to change notification settings - Fork 63
2017 07 05 Zipkin and X Ray interop
Created by Adrian Cole on Sep 17, 2018
We had a video conference to settle some details about X-Ray interop to avoid thrashing. Adrian represented Zipkin while a bunch of Amazon folks offered time for Q/A (Abhishek, Anssi, Bharath, and Davi)
Zipkin’s AWS repo has conversion mechanisms to go from less structured Zipkin v2 format to Amazon’s more structured format.
One concern is that applications routinely make intermediate spans between an entry and and exit, such as an incoming service call and an outgoing one. These intermediate spans need to be mapped to X-Ray, and notably we need to choose the correct name for these. We need to know the right approach so that things like dependency graphs show properly in the X-Ray UI.
Another is remote downstreams. Sometimes clients do not use discovery services, meaning the downstream can be remote and also unidentified. Typically, zipkin will either leave out a remote service name or call it “unknown”.
Untypically, there could be instrumentation bugs where a client (exit span) is a child of another client (exit span). In Zipkin’s dependency job, we address this by “skipping up the tree” essentially ignoring accidental layering of client spans. We need to figure out how to address this as notably in the X-Ray sender, we are usually sending a span at a time, not the whole trace or local view of one.
Here’s the code
if (span.kind() == null
|| span.kind() != Span.Kind.SERVER && span.kind() != Span.Kind.CONSUMER) {
writer.name("type").value("subsegment");
if (span.kind() != null) writer.name("namespace").value("remote");
// some remote service's name can be null, using null names causes invisible subsegment
// using "unknown" subsegment name will help to detect missing names
writer
.name("name")
.value(span.remoteServiceName() == null ? "unknown" : span.remoteServiceName());
} else {
writer.name("name").value(span.localServiceName());
}
https://github.com/openzipkin/zipkin-aws/pull/88 “Use local service name as X-Ray subsegment name”
If the intention is to not create a service link, don’t mark the subsegment as remote
For the remote name, we can have a fallback model: start with the remote service name, fallback to the hostname (if available), then the span name (because it is already hinted as low cardinality). When there is a problem, correct the remote service name.
There is still a problem when there is double instrumentation, because a remote will have a child of remote. Causing intermediate nodes to show up.
Document generated by Confluence on Jun 18, 2019 18:50