Not able to send messages to broker using PLAIN mechanism #538
Description
Describe the bug
I am trying to send messages from fluentd to kafka broker using PLAIN mechanism. I have configured username and password in the fluent.conf and set @type rdkafka2. The error message from fluentd says that security protocol must be correctly configured.
When I switch from @type rdkafka2 to @type kafka2, I am able to send messages successfully without any errors.
To Reproduce
I am using below config to send messages (with username and password)
<match kafka.message>
@type rdkafka2
brokers kf-test-kafka-0.kf-test-kafka-headless.preeti-krb-system.svc.cluster.local:9092
topic_key topic
default_topic fluentd-gzip
use_event_time true
username <<krb_principle>>
password <<krb_password>>
sasl_over_ssl true
compression_codec gzip
ssl_ca_cert /etc/kerberos/kafka/tls.crt
ssl_client_cert /etc/kerberos/kafka/tls.crt
ssl_client_cert_key /etc/kerberos/kafka/tls.key
ssl_client_cert_key_password test1234
<format>
@type json
</format>
<buffer topic>
@type file
path /var/log/fluentd-buffers/kafka
flush_mode interval
flush_interval 5s
chunk_limit_size 1m
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
log_level debug
</match>I have a kafka broker configured with SASL authentication using PLAIN mechanism.
Expected behavior
Ideally it is expected that when principal and keytab are passed then we are using GSSAPI mechanism, and if username and password is passed then it is PLAIN mechanism. But it appears that the plugin is not able to set the mechanism correctly and there is no way of configuring the mechanism by ourselves as the parameter is not exposed to users.
Your Environment
- Fluentd version: 1.16.9
- fluent-plugin-kafka version: 0.19.3
- ruby-kafka version: 1.5.0
- rdkafka: 0.12.0
- Operating system: Rocky 8
- Kernel version: 5.14Your Configuration
<match kafka.message>
@type rdkafka2
brokers kf-test-kafka-0.kf-test-kafka-headless.preeti-krb-system.svc.cluster.local:9092
topic_key topic
default_topic fluentd-gzip
use_event_time true
username <<krb_principle>>
password <<krb_password>>
sasl_over_ssl true
compression_codec gzip
ssl_ca_cert /etc/kerberos/kafka/tls.crt
ssl_client_cert /etc/kerberos/kafka/tls.crt
ssl_client_cert_key /etc/kerberos/kafka/tls.key
ssl_client_cert_key_password test1234
<format>
@type json
</format>
<buffer topic>
@type file
path /var/log/fluentd-buffers/kafka
flush_mode interval
flush_interval 5s
chunk_limit_size 1m
queue_limit_length 32
retry_max_interval 30
retry_forever true
</buffer>
log_level debug
</match>Your Error Log
2025-09-24 09:33:34 +0000 [warn]: #0 rdkafka: [thrd:app]: Configuration property `sasl.username` only applies when `sasl.mechanism` is set to PLAIN or SCRAM-SHA-..
2025-09-24 09:33:34 +0000 [warn]: #0 rdkafka: [thrd:ssl://kf-test-kafka-0.kf-test-kafka-headless.preeti-krb-system.]: ssl://kf-test-kafka-0.kf-test-kafka-headless.preeti-krb-system.svc.cluster.local:9092/bootstrap: Disconnected: verify that security.protocol is correctly configured, broker might require SASL authentication (after 383ms in state UP, 1 identical error(s) suppressed)Additional context
The first warning makes it pretty clear that the mechanism is somehow still set as GSSAPI instead of PLAIN. This same configuration works well with @type kafka2, is it an expected behavior?
Is there a parameter exposed to configure the mechanism and set it to PLAIN? or is it handled internally? If it is handled internally, then why is the plugin not able to change the mechanism to PLAIN despite the presence of username and password ?