Kafka receiver return "principal unknown"

[andreadna]

Component(s)

receiver/kafka

What happened?

Description

We configured kafka receiver as below:

receivers:
  kafka:
    brokers:
      - broker1:9093
      - broker2:9093
      - broker3:9093
      - broker4:9093
    topic: "monitoring-steps"
    protocol_version: 2.5.0
    group_id: "otel-kafka"
    encoding: "raw"
    client_id: "<username@realm>"
    auth:
      kerberos:
        service_name: "kafka"
        config_file: "/etc/krb5.conf"
        use_keytab: true
        username: "<username@realm>"
        keytab_file: "/opt/SP/home/kerberos_conf/username.keytab"
        realm: "<realm>"
        disable_fast_negotiation: true

but we receive an error as follow:

• [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database

Our Kerberos support SASL GSSAPI authentication with keytab. The same configuration (username, realm, keytab and krb5.conf) are used in other applications in the same server and they work fine.

Collector version

0.120.0

Environment information

Environment

OS: Linux 4.18.0-553.37.1.el8_10.x86_64

OpenTelemetry Collector configuration

receivers:
  kafka:
    brokers:
      - broker1:9093
      - broker2:9093
      - broker3:9093
      - broker4:9093
    topic: "monitoring-steps"
    protocol_version: 2.5.0
    group_id: "otel-kafka"
    encoding: "raw"
    client_id: "<username@realm>"
    auth:
      kerberos:
        service_name: "kafka"
        config_file: "/etc/krb5.conf"
        use_keytab: true
        username: "<username@realm>"
        keytab_file: "/opt/SP/home/kerberos_conf/username.keytab"
        realm: "<realm>"
        disable_fast_negotiation: true

Log output

2025-03-05T09:34:04.517Z        info    service@v0.120.0/service.go:193 Setting up own telemetry...
2025-03-05T09:34:04.517Z        info    builders/builders.go:26 Development component. May change in the future.        {"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "logs"}
2025-03-05T09:34:04.520Z        info    service@v0.120.0/service.go:258 Starting otelcol-contrib...     {"Version": "0.120.1", "NumCPU": 6}
2025-03-05T09:34:04.520Z        info    extensions/extensions.go:40     Starting extensions...
2025-03-05T09:34:05.476Z        error   graph/graph.go:426      Failed to start component       {"error": "kafka: client has run out of available brokers to talk to: 4 errors occurred:\n\t* [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database\n\t* [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database\n\t* [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database\n\t* [Root cause: KDC_Error] KDC_Error: AS Exchange Error: kerberos error response from KDC: KRB Error: (6) KDC_ERR_C_PRINCIPAL_UNKNOWN Client not found in Kerberos database\n", "type": "Receiver", "id": "kafka"}
2025-03-05T09:34:05.476Z        info    service@v0.120.0/service.go:323 Starting shutdown...
2025-03-05T09:34:05.479Z        info    extensions/extensions.go:68     Stopping extensions...
2025-03-05T09:34:05.479Z        info    service@v0.120.0/service.go:337 Shutdown complete.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

Client not found in Kerberos database seems to indicate that the username is not found. Have you tried without adding @realm to it by any chance?

[atoulme]

Further, is there a way for the codeowners to reproduce this issue? Is it something you'd be able to offer as an integration test?

[andreadna]

Client not found in Kerberos database seems to indicate that the username is not found. Have you tried without adding @realm to it by any chance?

I tried without "realm" and i receive error because it's needed. About the username, it exists because the same kerberos configuration are used in one other application on the same server and it works.
I tried to combine also TLS settings adding CA but no way.