"otlp_proto" Encoding is Not Working but "otlp_json" Works (but is marked as experimental)

[janFlueckiger]

Component(s)

exporter/kafka

Describe the issue you're reporting

Hello 🖖

I wanted to export metrics, traces, and logs from my OpenTelemetry Collector to a Kafka Topic. And during my research, I found this kafkaexporter. And it works very well. But I have some questions regarding the encoding for metrics.

I am using "otlp_json" but it mentions that it is EXPERIMENTAL. It works perfectly fine, but since I want to use it in a productive environment EXPERIMENTAL is not something I am looking for. But when I use "otlp_proto" (where nothing states to be experimental), the content of the Kafka-Topic is in binary and completely unreadable.

Encoding: "otlp_json" - Readable, and when I open it with VSCode it can be formatted into JSON format

Encoding: "otlp_proto" - Completely unreadable

This is the OpenTelemetry Collector

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
    exporters:
      kafka/metrics:
        protocol_version: 6.1.0
        brokers: kafka.kafka:9092
        topic: otelcol-metrics
        encoding: otlp_json
      kafka/traces:
        protocol_version: 6.1.0
        brokers: kafka.kafka:9092
        topic: otelcol-traces
        encoding: jaeger_json
      kafka/logs:
        protocol_version: 6.1.0
        brokers: kafka.kafka:9092
        topic: otelcol-logs
        encoding: raw
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [kafka/traces, logging]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [kafka/metrics, logging]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [kafka/logs, logging]

So here to my questions:

1. Is it a known issue that "otlp_proto" does not work as encoding, or is the README maybe not up to date?
2. Will the development of the "otlp_json" encoding be continued so it can be used in a productive environment?
3. If it is planned to finish "otlp_json" will it be very different to the current version? Or what will be the biggest differences?

Thank you very much in advance!

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[pavolloffay]

Is it a known issue that "otlp_proto" does not work as encoding, or is the README maybe not up to date?

Do you see any other issues than otlp_proto into being readable? It's a protobuf is a binary protocol, it's not readable text like JSON.

Will the development of the "otlp_json" encoding be continued so it can be used in a productive environment?
If it is planned to finish "otlp_json" will it be very different to the current version? Or what will be the biggest differences?

It's a question for the specification repository https://github.com/open-telemetry/opentelemetry-specification or slack channel. The kafka integration in the collector just uses provided formats.

[mx-psi]

As of open-telemetry/opentelemetry-specification/pull/2930 OTLP/JSON format is considered stable, we should probably remove the 'experimental' notice

[janFlueckiger]

Okay, thank you very much for your help!

[govardhanda-Kore]

Is the JSON payload in Kafka receiver for hotel collector accepted, or are there any feature plans to include it?