tag/span.kind server/client value seems to be reversed when processing Zipkin V1 json

[apm-opentt]

Requirement - what kind of business use case are you trying to solve?

I am testing sending Zipkin V1 and V2 span json to Jaeger Collector via http. Jaeger converts the Zipkin span json to Jaeger format and stores that in Cassandra. I expected the converted json in Jaeger format to be consistent in meaning for both V1 and V2 Zipkin json.

Problem - what in Jaeger blocks you from solving the requirement?

Consider the scenario service A calls service B. One would expect A's span.kind value to be client and B's span.kind value to be server. Indeed when I send a V2 Zipkin json, this is true. However, When I send a V1 Zipkin json the Jaeger converted span has A's span.kind equal to server and B's span.kind equal to client.
Note: For V1, I posted to /api/v1/spans endpoint and for V2 I posted to /api/v2/spans endpoint.

You can reproduce this issue by sending the V1 Zipkin json from https://zipkin.io/pages/data_model.html. Here is the converted span produced by Jaeger. Can you help confirm if this is an issue? If not, can you help explain why V1 Zipkin spans are converted this way?

{
  "data": [
    {
      "processes": {
        "p1": {
          "serviceName": "zipkin-query",
          "tags": [
            {
              "key": "ip",
              "type": "string",
              "value": "192.168.1.2"
            }
          ]
        }
      },
      "spans": [
        {
          "duration": 386000,
          "logs": [],
          "operationName": "get",
          "processID": "p1",
          "references": [],
          "spanID": "bd7a977555f6b982",
          "startTime": 1458702548467000,
          "tags": [
            {
              "key": "span.kind",
              "type": "string",
              "value": "server"
            },
            {
              "key": "internal.span.format",
              "type": "string",
              "value": "zipkin"
            }
          ],
          "traceID": "bd7a977555f6b982",
          "warnings": null
        },
        {
          "duration": 13000,
          "logs": [],
          "operationName": "query",
          "processID": "p1",
          "references": [
            {
              "refType": "CHILD_OF",
              "spanID": "ebf33e1a81dc6f71",
              "traceID": "bd7a977555f6b982"
            }
          ],
          "spanID": "be2d01e33cc78d97",
          "startTime": 1458702548786000,
          "tags": [
            {
              "key": "jdbc.query",
              "type": "string",
              "value": "select distinct `zipkin_spans`.`trace_id` from `zipkin_spans` join `zipkin_annotations` on (`zipkin_spans`.`trace_id` = `zipkin_annotations`.`trace_id` and `zipkin_spans`.`id` = `zipkin_annotations`.`span_id`) where (`zipkin_annotations`.`endpoint_service_name` = ? and `zipkin_spans`.`start_ts` between ? and ?) order by `zipkin_spans`.`start_ts` desc limit ?"
            },
            {
              "key": "peer.service",
              "type": "string",
              "value": "spanstore-jdbc"
            },
            {
              "key": "peer.ipv4",
              "type": "string",
              "value": "127.0.0.1"
            },
            {
              "key": "peer.port",
              "type": "int64",
              "value": 3306
            },
            {
              "key": "span.kind",
              "type": "string",
              "value": "client"
            },
            {
              "key": "internal.span.format",
              "type": "string",
              "value": "zipkin"
            }
          ],
          "traceID": "bd7a977555f6b982",
          "warnings": null
        },
        {
          "duration": 354374,
          "logs": [],
          "operationName": "get-traces",
          "processID": "p1",
          "references": [
            {
              "refType": "CHILD_OF",
              "spanID": "bd7a977555f6b982",
              "traceID": "bd7a977555f6b982"
            }
          ],
          "spanID": "ebf33e1a81dc6f71",
          "startTime": 1458702548478000,
          "tags": [
            {
              "key": "component",
              "type": "string",
              "value": "JDBCSpanStore"
            },
            {
              "key": "request",
              "type": "string",
              "value": "QueryRequest{serviceName=zipkin-query, spanName=null, annotations=[], binaryAnnotations={}, minDuration=null, maxDuration=null, endTs=1458702548478, lookback=86400000, limit=1}"
            },
            {
              "key": "internal.span.format",
              "type": "string",
              "value": "zipkin"
            }
          ],
          "traceID": "bd7a977555f6b982",
          "warnings": null
        },
        {
          "duration": 1000,
          "logs": [],
          "operationName": "query",
          "processID": "p1",
          "references": [
            {
              "refType": "CHILD_OF",
              "spanID": "ebf33e1a81dc6f71",
              "traceID": "bd7a977555f6b982"
            }
          ],
          "spanID": "13038c5fee5a2f2e",
          "startTime": 1458702548817000,
          "tags": [
            {
              "key": "jdbc.query",
              "type": "string",
              "value": "select `zipkin_spans`.`trace_id`, `zipkin_spans`.`id`, `zipkin_spans`.`name`, `zipkin_spans`.`parent_id`, `zipkin_spans`.`debug`, `zipkin_spans`.`start_ts`, `zipkin_spans`.`duration` from `zipkin_spans` where `zipkin_spans`.`trace_id` in (?)"
            },
            {
              "key": "peer.service",
              "type": "string",
              "value": "spanstore-jdbc"
            },
            {
              "key": "peer.ipv4",
              "type": "string",
              "value": "127.0.0.1"
            },
            {
              "key": "peer.port",
              "type": "int64",
              "value": 3306
            },
            {
              "key": "span.kind",
              "type": "string",
              "value": "client"
            },
            {
              "key": "internal.span.format",
              "type": "string",
              "value": "zipkin"
            }
          ],
          "traceID": "bd7a977555f6b982",
          "warnings": null
        },
        {
          "duration": 2000,
          "logs": [],
          "operationName": "query",
          "processID": "p1",
          "references": [
            {
              "refType": "CHILD_OF",
              "spanID": "ebf33e1a81dc6f71",
              "traceID": "bd7a977555f6b982"
            }
          ],
          "spanID": "37ee55f3d3a94336",
          "startTime": 1458702548827000,
          "tags": [
            {
              "key": "jdbc.query",
              "type": "string",
              "value": "select `zipkin_annotations`.`trace_id`, `zipkin_annotations`.`span_id`, `zipkin_annotations`.`a_key`, `zipkin_annotations`.`a_value`, `zipkin_annotations`.`a_type`, `zipkin_annotations`.`a_timestamp`, `zipkin_annotations`.`endpoint_ipv4`, `zipkin_annotations`.`endpoint_port`, `zipkin_annotations`.`endpoint_service_name` from `zipkin_annotations` where `zipkin_annotations`.`trace_id` in (?) order by `zipkin_annotations`.`a_timestamp` asc, `zipkin_annotations`.`a_key` asc"
            },
            {
              "key": "peer.service",
              "type": "string",
              "value": "spanstore-jdbc"
            },
            {
              "key": "peer.ipv4",
              "type": "string",
              "value": "127.0.0.1"
            },
            {
              "key": "peer.port",
              "type": "int64",
              "value": 3306
            },
            {
              "key": "span.kind",
              "type": "string",
              "value": "client"
            },
            {
              "key": "internal.span.format",
              "type": "string",
              "value": "zipkin"
            }
          ],
          "traceID": "bd7a977555f6b982",
          "warnings": null
        }
      ],
      "traceID": "bd7a977555f6b982",
      "warnings": null
    }
  ],
  "errors": null,
  "limit": 0,
  "offset": 0,
  "total": 0
}

Proposal - what do you suggest to solve the problem or improve the existing situation?

Any open questions to address

[pavolloffay]

This indeed seems like a bug. I will have a look at it.

[pavolloffay]

@apm-opentt could you please paste here Zipkin V1 JSON with client and server span which is causing the issue?

[pavolloffay]

I wasn't able to reproduce this. This is zipkin spans v1:

spans.txt

UI spans
ui-spans.txt

Data from ES:
es-spans.txt

[apm-opentt]

@pavolloffay Thanks for looking into this issue. I used your spans.txt and I did get the same output as your ui-spans.txt result. However, if I used this Zipkin V1 json I am getting the span.kind reversed.
input json:
zipkin-v1-AtoB.txt
output json:
converted-AtoB.txt
Also, if you copy and paste the sample json from the official Zipkin V1 data model page and send it to Jaeger the converted json will have the same issue too. https://zipkin.io/pages/data_model.html

[apm-opentt]

@pavolloffay any updates on this defect? Thanks.

[pavolloffay]

I do not see any error in the data you attached. The span.kind is correctly added. The cs, cr span is client and ss, sr is server. Also consider using distinct service names. Both services in your example have the same service name (unless the service is calling itself).

cs 376e1284-d2a1-41a1-86b0-b23677919591_0bc2b74be0544f36c57313e8843c4e04_default_k8sService_acmeair3 d39e934a53c9b551
cr 376e1284-d2a1-41a1-86b0-b23677919591_0bc2b74be0544f36c57313e8843c4e04_default_k8sService_acmeair3
sr 376e1284-d2a1-41a1-86b0-b23677919591_0bc2b74be0544f36c57313e8843c4e04_default_k8sService_acmeair3 f01c88a51e90614f 10.1.3.106
ss 376e1284-d2a1-41a1-86b0-b23677919591_0bc2b74be0544f36c57313e8843c4e04_default_k8sService_acmeair3                  10.1.3.106

[pavolloffay]

I am closing this as it does not seem to be an issue. Feel free to comment/re-open if there is any other issues.

[apm-opentt]

@pavolloffay Here is the screen shot on Jaeger UI for the zipkin-v1-AtoB.txt https://github.com/jaegertracing/jaeger/files/4192948/zipkin-v1-AtoB.txt. It shows the span.kind being reversed. Can you help take a look?

[apm-opentt]

Hi @pavolloffay, did you get a chance to look at the screen capture above ^^. It looks like a defect to me. What do you think? If it is defect, can you reopen this issue? Thanks.

[yurishkuro]

It's not a defect in Jaeger, but in whichever instrumentation you're using. Specifically, in your screenshot the parent span ID is f01c88.... The JSON for that span has this:

  {
    "traceId": "eeb91b2cf40c808af01c88a51e90614f",
    "id": "f01c88a51e90614f",
    "name": "/search/456",
    "timestamp": 1581316483691000,
    "duration": 120845,
    "annotations": [
      {
        "value": "sr",

As you can see, it's reporting itself as a "server" span, via "sr" (Server Receive) annotation. Jaeger is simply translating this into span.kind=server.

[apm-opentt]

@yurishkuro, Thanks for helping to look at this issue. It is fair to question the source and validity of the json I posted to Jaeger collector's http endpoint in my previous comments. So let me use a json from an authoritative source. I am now posting the json from Zipkin's official page for V1 data format. https://zipkin.io/pages/data_model.html. I simply copied and paste the json in this page and send via http to Jaeger collect's v1 endpoint. Here is what Jaeger UI shows. One would expect the parent spans to have span.kind == client and children spans to have span.kind == server. It would be consistent with the behavior for spans that came from Jaeger agents. However, as you can see in screen capture the span.kind is reversed for Zipkin v1 json. (Note: Zipkin v2 json produced spans are consistent with spans from Jaeger agents).

I think the "sr", "ss", "cs" and "cr" in Zipkin v1 are not very intuitive and I think it actually means the following.

[yurishkuro]

The last screenshot looks fine to me, while your red annotations are incorrect (and don’t match your diagram).

[pavolloffay]

@pavolloffay Here is the screen shot on Jaeger UI for the zipkin-v1-AtoB.txt https://github.com/jaegertracing/jaeger/files/4192948/zipkin-v1-AtoB.txt. It shows the span.kind being reversed. Can you help take a look?

I think it is correcnt because span with cs, cr have set parentSpanId to a span with ss and sr annotations.

[apm-opentt]

@pavolloffay @yurishkuro Thank you for patiently answering my questions. This may or may not be an issue, but I would like to understand why the span.kind value is different with different input json. I was under the impression that the root span should always have tag span.kind=client. Here I am sending the spans.txt from Pavol's comment to this issue #2067 (comment) and here is the resulting Jaeger UI screen capture. You can see that the root span has span.kind=client and child span has span.kind=server which is different than the screen capture in my previous post. Can you help explain the inconsistency?

[yurishkuro]

Jaeger does not make decisions which span is client or server, it simply renders what instrumentation reports. The correct value for instrumentation to use depends on the role of the particular span - if you're making an outbound call, the span should be "client" kind. It has no relation to being root or not - root is simply where the trace originates, which can be in either states.