Kafka Ingester

[davit-y]

Requirement - what kind of business use case are you trying to solve?

An Ingester allows users to queue spans in Kafka instead of writing directly to storage in Collector. The Ingester consumes spans from a configurable Kafka topic and writes to any number of desired storage types.

Problem - what in Jaeger blocks you from solving the requirement?

This kind of queueing functionality will be new to Jaeger.

Checklist

• Initial code Move over internal ingester code to OSS #940
• Split Kafka config into Producer/Consumer Add producer folder to pkg/kafka/config #957
• Changes to consumer Ingester Consumer #942
• Changes to processor Ingester Processor #944
• Create builder, flags, main Ingester Main #952
• Integration test Ingester integration test #968
• Docker image Ingester binaries & Docker #1086
• Pre-built binary Publish binaries for Kafka ingesters #1031
• Docs / instructions / examples how to run Add documentation for Kafka ingestion #1030

[yurishkuro]

We should also be able to specify the format of the spans, to allow us to accept Zipkin spans.

[yurishkuro]

Are we planning to support multiple writers in the ingester? If so, you may need to address separate Kafka reader/writer configurability. Is there a plan how it will be done?

[davit-y]

Yes, we will support multiple writers. The configuration for the writers will use the current storage factory configuration flags. For the consumer we will have separate Kafka/consumer options (Brokers,Topic,GroupID,Parallelism) similar to the CollectorOptions.

[yurishkuro]

I added a couple more tasks

[yurishkuro]

I created https://hub.docker.com/r/jaegertracing/jaeger-ingester/

[Dieterbe]

Hello,
this sounds interesting to me. Since we are looking to process spans with some custom logic and analyze traces to generate reports.
I know we could probably already do something like this using flink, but we're new to that, whereas we're quite familiar with kafka.

so what's the status of this?

[black-adder]

If all you care about is reading spans from kafka directly, you can configure the collector to dual write to cassandra and kafka. We need to cut a new release to enable that feature but it should already be available in master.

[pavolloffay]

The last missing part is documentation. Which is tracked separately #1030

[bohea]

I know this issue was closed.
but if jaeger-ingester consuming spans from kafka
who is playing role of producer sending spans to kafka, is jaeger-agent or jaeger-client?

[vprithvi]

In the current setup, jaeger-collector sends spans to Kafka