Kafka receiver and exporter

[pavolloffay]

Is your feature request related to a problem? Please describe.

Use Kafka as an intermediary buffer between collectors and exporter to an actual storage/vendor.

This issue is a possible duplicate of open-telemetry/opentelemetry-collector-contrib#268 and open-telemetry/opentelemetry-collector-contrib#5.

Should be Kafka supported in the contrib or in this repository if there is substantial interest?

Other references to Kafka related issues:

• Make OTel Collector serve Jaeger Remote Sampling directly #500 (comment)
• Add Support for Jaeger Kafka Exporter opentelemetry-collector-contrib#125
• pdata.TracesFromOtlp can't be used externally #1101

Describe the solution you'd like

Kafka receiver and exporter hosted in OpenTelemetry collector.

Describe alternatives you've considered

Host the implementation in the Jaeger project.

Additional context

None.

[bogdandrutu]

@pavolloffay definitely we can host this, some questions:

• One of the issue mentioned to support Jaeger Kafka, I think you are asking for OTLP Kafka, am I correct?
• How many dependencies do we bring if we put this in core? Just a rough estimate based on your experience.

[pavolloffay]

One of the issue mentioned to support Jaeger Kafka, I think you are asking for OTLP Kafka, am I correct?

Primarily we should support OTLP in this repository. However supporting additional formats that are already imported (Jaeger, Zipkin) in this project could help users with migration.

How many dependencies do we bring if we put this in core? Just a rough estimate based on your experience.

It depends on the chosen Kafka client. There are a couple of options:

• https://github.com/Shopify/sarama (used by Jaeger with deprecated https://github.com/bsm/sarama-cluster and supported alternative with some issues consumerGroup will deadlock when handling errors IBM/sarama#1554 and why example consumergroup can not work correctly ? only one client can consume message, other client can not work IBM/sarama#1516)
• https://github.com/segmentio/kafka-go
• https://github.com/confluentinc/confluent-kafka-go - a wrapper around native librdkafka C client. This would make the build and portability more difficult.

@yurishkuro you have more experience with the Sarama client. Could you please share your preference on the driver?

[pavolloffay]

From the number of 3rd party dependencies the https://github.com/segmentio/kafka-go/blob/master/go.mod seems to be better than https://github.com/Shopify/sarama/blob/master/go.mod

[pavolloffay]

The kafka-go does not support GSSAPI/Kerberos segmentio/kafka-go#237 (though, we could contribute it)

[yurishkuro]

@pavolloffay we've had some issue with sarama when partition rebalancing happens (you may recall the "deadlock detector"). But we're not on the latest version and should consider upgrading. I don't have any experience with sementio or confluence drivers. The latter, as you said, requires cgo, which we so far tried to avoid.

[objectiser]

Maybe the best option is to stick with sarama for now (as a known quantity). Sarama "currently officially support .... Kafka 2.3 through 2.5" whereas sementio/kafka-go "is currently compatible with Kafka versions from 0.10.1.0 to 2.1.0".

Although both would probably work fine with latest Kafka, it seems to suggest sarama is more closely tracking the Kafka releases.

[pavolloffay]

I am also more leaning toward sarama since it supports also Kerberos.