how to search annotations or binaryAnnotations in kibana? #1412

Open
dragontree101 opened this Issue Nov 20, 2016 · 10 comments

Projects

None yet

4 participants

@dragontree101

i send trace info to es5, and i want to search annotations or binaryAnnotations from kibana.

i have one span json like this:

{
  "_index": "zipkin-2016-11-17",
  "_type": "span",
  "_id": "AVhw8k-GQpNFwf9ty7hN",
  "_score": null,
  "_source": {
    "traceId": "0f56baad2910df69",
    "timestamp_millis": 1479363860752,
    "annotations": [
      {
        "endpoint": {
          "ipv4": "10.8.0.2",
          "port": 18088,
          "serviceName": "first-example-dev"
        },
        "value": "cs",
        "timestamp": 1479363860752000
      },
      {
        "endpoint": {
          "ipv4": "10.8.0.2",
          "port": 18088,
          "serviceName": "first-example-dev"
        },
        "value": "cr",
        "timestamp": 1479363860903752
      }
    ],
    "parentId": "20c251aac430bf39",
    "duration": 151752,
    "binaryAnnotations": [
      {
        "endpoint": {
          "ipv4": "10.8.0.2",
          "port": 19091,
          "serviceName": "thrift"
        },
        "value": true,
        "key": "sa"
      }
    ],
    "name": "secondadd",
    "collector_timestamp_millis": "2016-11-17T06:21:18.086+0000",
    "id": "f83891ef110158ef",
    "timestamp": 1479363860752000
  },
  "fields": {
    "timestamp_millis": [
      1479363860752
    ],
    "collector_timestamp_millis": [
      1479363678086
    ]
  },
  "sort": [
    1479363678086
  ]
}

i want to search annotations's endpoint's serviceName is first-example-dev, and i direct search first-example-dev from kibana, no result :(, and i search annotations:first-example-dev also nothing.

how can i search this content?

@adriancole
Contributor
@adriancole
Contributor

just in case.. @openzipkin/elasticsearch does anyone happen to know what this might imply? I know that in zipkin itself annotation searches are nested terms queries

@mansu
Contributor
mansu commented Nov 21, 2016

@adriancole I think @dragontree101 is asking for example ES queries on how to search for spans from the kibana UI. I think this is a valid use case that we should document since most people who use ES will also use Kibana. In practice, I found that Kibana is a better UI to search for traces esp when they are broken and don't show up the UI. We are currently doing the same, happy to share what we come up with shortly.

@adriancole
Contributor

@mansu well the point in suggesting reliable channels for Kibana questions isn't a question about if it is a valid use case, or not. Rather, we haven't yet established anything near reliability for a place to come to for Kibana support. We neither document, run or test it for starters! Only thing we have right now is a note about timestamps (and an ad-hoc test I did once to see if maybe it worked or not). Even if we did support Kibana, we'd want the community to know where to get more help, or help when one of those who know Kibana aren't looking.

Anyway...

I'm glad you're interested in helping, as that'd be the first step towards supporting something sustainably. If you come up with something, let's put it in the README in a section for Kibana. Then, maybe @bsideup and/or I can help make sure it is tested. That way, we don't accidentally break it, which would be sad.

Sound good!

@dragontree101

thanks @adriancole and @mansu
i both use zipkin's ui and kibana ui, i think zipkin's ui find one trace very good.
but i put some other import info into annotation or binaryAnnotation, so i want to aggregation some annotation or binaryAnnotation 's key or value in spans in kibana, because of i think kibana's aggregation better than zipkin's ui.
:)

@adriancole
Contributor

well, it could only be better if you use elasticsearch :) ps whatever isn't good should at least be noted for improvement in with a screen shot of what's better (since good is subjective and not always guessable) https://github.com/openzipkin/zipkin-ui/issues

Anyway, in case it helps, here's a complex query that the http implementation sends on the wire. It might help you figure out what to do (or not).

ex.

QueryRequest.builder().serviceName("service")
    .spanName("methodcall")
    .addAnnotation("custom")
    .addBinaryAnnotation("BAH", "BEH").build();

becomes this http request:

POST zipkin/span/_search?allow_no_indices=true&expand_wildcards=open&ignore_unavailable=true
...

{
  "size" : 0,
  "query" : {
    "bool" : {
      "must" : {
        "match_all" : { }
      },
      "filter" : {
        "bool" : {
          "must" : [ {
            "range" : {
              "timestamp_millis" : {
                "from" : 0,
                "to" : 1479693481467,
                "include_lower" : true,
                "include_upper" : true
              }
            }
          }, {
            "bool" : {
              "should" : [ {
                "nested" : {
                  "query" : {
                    "term" : {
                      "annotations.endpoint.serviceName" : "service"
                    }
                  },
                  "path" : "annotations"
                }
              }, {
                "nested" : {
                  "query" : {
                    "term" : {
                      "binaryAnnotations.endpoint.serviceName" : "service"
                    }
                  },
                  "path" : "binaryAnnotations"
                }
              } ]
            }
          }, {
            "term" : {
              "name" : "methodcall"
            }
          }, {
            "nested" : {
              "query" : {
                "bool" : {
                  "must" : [ {
                    "term" : {
                      "annotations.value" : "custom"
                    }
                  }, {
                    "term" : {
                      "annotations.endpoint.serviceName" : "service"
                    }
                  } ]
                }
              },
              "path" : "annotations"
            }
          }, {
            "nested" : {
              "query" : {
                "bool" : {
                  "must" : [ {
                    "term" : {
                      "binaryAnnotations.key" : "BAH"
                    }
                  }, {
                    "term" : {
                      "binaryAnnotations.value" : "BEH"
                    }
                  }, {
                    "term" : {
                      "binaryAnnotations.endpoint.serviceName" : "service"
                    }
                  } ]
                }
              },
              "path" : "binaryAnnotations"
            }
          } ]
        }
      }
    }
  },
  "aggregations" : {
    "traceId_agg" : {
      "terms" : {
        "field" : "traceId",
        "size" : 10,
        "order" : {
          "timestamps_agg" : "desc"
        }
      },
      "aggregations" : {
        "timestamps_agg" : {
          "min" : {
            "field" : "timestamp_millis"
          }
        }
      }
    }
  }
}
@sammypbaird

@adriancole , thanks for the sample elasticsearch query. I was having trouble querying the serviceName too, until I realized from your code snippet that the annotations are stored as a nested object in elasticsearch, which requires a different type of query. One question I had though was why the serviceName is stored for every annotation. I would assume serviceName is applicable to the whole span, and should be stored as a field on the span. Under what situations can you have different serviceNames for the same span?

@dragontree101 , here's a simpler query that you can use on elasticsearch to find spans that match a specific serviceName (in this example, I'm searching for spans with a serviceName of jdbc):

{
  "query": {
    "bool": {
      "should": [
        {
          "nested": {
            "query": {
              "term": {
                "annotations.endpoint.serviceName": "jdbc"
              }
            },
            "path": "annotations"
          }
        }
      ]
    }
  }
}

That is what worked for me.

@adriancole
Contributor
@sammypbaird

Sorry, I meant the raw spans (i.e. the spans you get when you query a trace from the zipkin api and pass in the parameter "raw"). Each raw span looks like they always have the same serviceName (in the raw trace, the client and server belong to different spans). I would imagine that the serviceName would exist at the same level as the span name, but instead it's populated for each annotation in the raw span, which makes querying from elasticsearch more difficult. I'm sure there is a reason for this, just wondering why.

@adriancole
Contributor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment