Failed to initialize SASL authentication:generic failure (after 0ms in state AUTH_REQ)

[haolujun]

Description

sasl_plaintext://kafka001.bx.momo.com:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ)
% ERROR: Local: Authentication failure: sasl_plaintext://kafka001.bx.momo.com:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ)
% ERROR: Local: All broker connections are down: 1/1 brokers are down : terminating

How to reproduce

echo "abc" | ./kafkacat -P -b ${IP}:${PORT} -t "test" -F kafka.conf
ecurity.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.mechanisms=GSSAPI
sasl.kerberos.keytab=recommend.keytab
sasl.kerberos.principal=recommend@${REALM}
sasl.kerberos.service.name=kafka

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.6.0-1-gc6d4b5
• Apache Kafka version: 1.0.0
• librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
• Operating system: Linux 3.10.0-693.21.1.el7.x86_64 Bugfix in example code #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Provide logs (with debug=.. as necessary) from librdkafka
  % Reading configuration from file kafka.conf
  %7|1595934395.505|SASL|rdkafka#producer-1| [thrd:app]: Selected provider Cyrus for SASL mechanism GSSAPI
  %7|1595934395.505|SASLREFRESH|rdkafka#producer-1| [thrd:app]: Refreshing Kerberos ticket with command: kinit -R -t "/home/xxxx/depend_libs/kafkacat/bin/recommend.keytab" -k recommend@${REALM} || kinit -t "/home/xxxx/depend_libs/kafkacat/bin/recommend.keytab" -k recommend@${REALM}
  %7|1595934395.513|SASLREFRESH|rdkafka#producer-1| [thrd:app]: Kerberos ticket refreshed in 8ms
  %7|1595934395.513|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.5.0-dirty (0x10500ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,sasl_oauthbearer, GCC GXX PKGCONFIG INSTALL GNULD LDS LIBDL PLUGINS ZLIB SSL SASL_CYRUS HDRHISTOGRAM SYSLOG SNAPPY SOCKEM SASL_SCRAM SASL_OAUTHBEARER CRC32C_HW, debug 0x200)
  %7|1595934395.516|PROTOERR|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1945) (incorrect broker.version.fallback?)
  %7|1595934395.516|PROTOERR|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: ApiArrayCnt -1 out of range
  %7|1595934395.518|AUTH|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
  %7|1595934395.519|SASLMECHS|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker supported SASL mechanisms: GSSAPI
  %7|1595934395.519|AUTH|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
  %7|1595934395.519|SASL|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Initializing SASL client: service name kafka, hostname ${broker}, mechanisms GSSAPI, provider Cyrus
  %3|1595934395.519|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ)
  % ERROR: Local: Authentication failure: sasl_plaintext://${broker}:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ)
  % ERROR: Local: All broker connections are down: 1/1 brokers are down : terminating
• Provide broker log excerpts
  %7|1595934168.382|CONNECTED|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Connected (Bugfix in example code #1)
  %7|1595934168.382|FEATURE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
  %7|1595934168.382|STATE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
  %7|1595934168.386|PROTOERR|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Protocol parse failure for ApiVersion v3 at 3/6 (rd_kafka_handle_ApiVersion:1945) (incorrect broker.version.fallback?)
  %7|1595934168.386|PROTOERR|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: ApiArrayCnt -1 out of range
  %7|1595934168.386|APIVERSION|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
  %7|1595934168.388|FEATURE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,SaslAuthReq
  %7|1595934168.388|AUTH|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Auth in state APIVERSION_QUERY (handshake supported)
  %7|1595934168.388|STATE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE
  %7|1595934168.389|SASLMECHS|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker supported SASL mechanisms: GSSAPI
  %7|1595934168.389|AUTH|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Auth in state AUTH_HANDSHAKE (handshake supported)
  %7|1595934168.389|STATE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ
  %7|1595934168.389|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ) (_AUTHENTICATION)
  %3|1595934168.389|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ)
  %7|1595934168.389|STATE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker changed state AUTH_REQ -> DOWN
  % ERROR: Local: Authentication failure: sasl_plaintext://${broker}:9092/bootstrap: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ)
  %7|1595934168.389|CONNECT|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: Not selecting any broker for cluster connection: still suppressed for 41ms: broker down
  % ERROR: Local: All broker connections are down: 1/1 brokers are down : terminating
  %7|1595934168.389|STATE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker changed state DOWN -> INIT
  %7|1595934168.389|STATE|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
  %7|1595934168.389|RECONNECT|rdkafka#producer-1| [thrd:sasl_plaintext://${broker}:9092/bootstrap]: sasl_plaintext://${broker}:9092/bootstrap: Delaying next reconnect by 162ms
• Critical issue

[haolujun]

After many test, I found that env were different between two machines got differnet response , And I got an error on failed machine: Failed to initialize SASL authentication: generic failure (after 0ms in state AUTH_REQ) (_AUTHENTICATION): identical to last error: error log suppressed.