Multi binder Multi Cluster kerberos jaas configuration fails with KRBError

[winkidzz]

Environment:

Multi binder kerberos setup as explained at https://kafka.apache.org/documentation/#security_client_staticjaas

Spring Configuration:

binders:
    kafka1:
      type: kafka
      environment:
         spring:
           cloud:
             stream:
              kafka:
                binder.brokers: broker1:port
                binder.jaas.loginModule: com.sun.security.auth.module.Krb5LoginModule
                binder.configuration.sasl.kerberos.service.name: svc_name_1
                binder.configuration.security.protocol: SASL_SSL
                binder.configuration.ssl.truststore.type: JKS
                binder.configuration.ssl.truststore.location: [hidden]
                binder.configuration.ssl.truststore.password: [hidden]
    kafka2:
      type: kafka
      environment:
        spring:
          cloud:
            stream:
              kafka:
                binder.brokers: broker2:port
                binder.jaas.loginModule: com.sun.security.auth.module.Krb5LoginModule
                binder.configuration.sasl.kerberos.service.name: svc_name_2
                binder.configuration.security.protocol: SASL_SSL
                binder.configuration.ssl.truststore.type: JKS
				binder.configuration.ssl.truststore.location: [hidden]
                binder.configuration.ssl.truststore.password: [hidden]

Error:

Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject

Credentials serviceCredsSingle: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23.
EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
getKDCFromDNS using UDP
KrbKdcReq send: kdc=[hidden]. TCP:88, timeout=30000, number of retries =3, #bytes=1530
KDCCommunication: kdc=[hidden]. TCP:88, timeout=30000,Attempt =1, #bytes=1530
DEBUG: TCPClient reading 125 bytes
KrbKdcReq send: #bytes read=125
KdcAccessibility: remove [hidden].:88
KDCRep: init() encoding tag is 126 req type is 13
KRBError:
sTime is Thu Jun 25 15:11:36 EDT 2020 1593112296000
suSec is 187798
error code is 7
error Message is Server not found in Kerberos database
sname is svc_name_1/broker2
msgType is 30

Findings:

The service name from the first broker stays for all binders. The service name of the second binder should replace the first one.

[winkidzz]

I am able to successfully replicate this issue in the kafka-multibinder-jaas example version and in the latest version of scst

[winkidzz]

Additional Information:

While bootstrapping the binders, KafkaTopicProvisioner.createAdminClient() second binder properties are set correctly

this.adminClientProperties = {HashMap@7532} size = 8
"security.protocol" -> "SASL_SSL"
"ssl.truststore.location" -> "cacerts"
"sasl.mechanism" -> "GSSAPI"
"bootstrap.servers" -> "broker2:port"
"ssl.truststore.password" -> "changeit"
"ssl.truststore.type" -> "JKS"
"sasl.kerberos.service.name" -> "svc_name_2"
"ssl.endpoint.identification.algorithm" -> ""

ChannelBuilders in kafka-clients library loads the default KafkaClient jaasContext which has the first principal name svc_name_2

[winkidzz]

Also note the jaas config information is sent using java property-Djava.security.auth.login.config=file:src\main\kafka_client_jaas.conf

[winkidzz]

Tested with spring boot parameter based configuration instead of java.security.auth.login.config. Since parameter based also creates temp file, the issue remains the same. The default service name KafkaClient from STATIC_INSTANCE kerberos is always the first one in the binder.

[winkidzz]

The LoginManager in kafka client jar caches the keytab contents and applies the servicename tied to cached content ignoring the currently passed configuration. In our case we use same keytab for both the brokers with different service name. hence the cached content always picks the first service name.

binders:
    kafka1:
      type: kafka
      environment:
         spring:
           cloud:
             stream:
              kafka:
                binder.brokers: broker1:9092
                binder.configuration.sasl.kerberos.service.name: svc_name_1
                binder.configuration.sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required
                  doNotPrompt=true
                  useKeyTab=true
                  useTicketCache=false
                  storeKey=true
                  keyTab="C:/keytab/keytab_1.keytab"
                  principal="principal_1@domain.COM";
                binder.configuration.security.protocol: SASL_SSL
                binder.configuration.ssl.truststore.type: JKS
                binder.configuration.ssl.truststore.location: /cacerts
                binder.configuration.ssl.truststore.password: pwd
    kafka2:
      type: kafka
      environment:
        spring:
          cloud:
            stream:
              kafka:
                binder.brokers: broker2:9092
                binder.configuration.sasl.kerberos.service.name: svc_name_2
                binder.configuration.sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required
                  doNotPrompt=true
                  useKeyTab=true
                  useTicketCache=false
                  storeKey=true
                  keyTab="C:/keytab/keytab_1.keytab"
                  principal="principal_1@domain.COM";
                binder.configuration.security.protocol: SASL_SSL
                binder.configuration.ssl.truststore.type: JKS
                binder.configuration.ssl.truststore.location: /cacerts
                binder.configuration.ssl.truststore.password: pwd

As jaas.config is same for both the brokers, svc_name_1 will be used to login to second broker causing failure.

Workaround:

1. Have separate key tabs. (Depending on your design this should be ideal solution).

2. Have same keytab with separate names

3. Hack the contents jaas.config to make it sligthly different without impacting lookup.

         Example: keyTab="C:/keytab/../keytab/keytab_1.keytab"
   

[sobychacko]

@winkidzz I think this issue is somewhat related to the same reasons raised in this issue: #874

The jaas configuration is set per classloader and the second one is ignored.

If the observations you made in this issue is for the same reasons as 874, then I am leaning towards the workarounds you suggested. Please let us know.

[winkidzz]

The class loader seems to be different issue. Here it is just matter of using same key tab across clusters and scs/kafka client not able to cache binder/servicename based dynamic caching support. It could be argued that each cluster should use different key tabs. It would benefit a lot if each binder's jaas configuration is supported independently.

[winkidzz]

I just re-read the #874. It is slightly different in terms of topic being in the same cluster. if the same key tab is used it would work because service name would be same for single cluster. The user is seeing error because of different key tabs been used and kafka client saslclienthandler is caching first binder configuration. having separate class loader could be a solution. It would help if apache kafka library can also be enhanced.

[sobychacko]

@winkidzz Did you get a chance to check with the Apache Kafka community to see how they handle this situation? i.e. the same JVM talking to two different Kafka clusters with the same configuration that you have. I think you will run into the same issues that you raised here. I am afraid, you might have to go with the workarounds you suggested. Please let us know. I don't see how we can address this cleanly from the binder without making those classloader level changes.

[winkidzz]

@sobychacko Let me reach out to them right away.

[sobychacko]

@winkidzz Any updates on this?

[winkidzz]

I have not received much attention via webchat. Created https://issues.apache.org/jira/browse/KAFKA-10276 couple of weeks back no movement yet.

[sobychacko]

@winkidzz Taking a look at this issue after a long time. It looks like Kafka KIP-85 was added to address this exact use case. According to the solutions implemented there, adding the following should be sufficient.

binder.configuration.sasl.jaas.config: com.sun.security.auth.module.Krb5LoginModule required
                  doNotPrompt=true
                  useKeyTab=true
                  useTicketCache=false
                  storeKey=true
                  keyTab="C:/keytab/keytab_1.keytab"
                  principal="principal_1@domain.COM";

You can use the same or different values for sasl.jaas.config in the second binder configuration based on KIP-85 implementation. Where do you see that the service name gets cached from the first binder configuration?
Also, I am curious how this workaround worked: keyTab="C:/keytab/../keytab/keytab_1.keytab". If it worked, that means that if you change the content of the jaas.config, then Kafka's auth mechanism will treat this as a new config. This fact is also in line with changing the name of the keytab file (your workaround-2). In addition, you said that if you used a separate keytab for each cluster (workaround - 1), that will also address the problem - correct? If so, in that scenario, would you use the same keytab file name? If you can chime in on this and comment, we can hopefully close this long-standing issue.

Thanks!

[sobychacko]

@winkidzz Please see the updates on these issues.

#874 (comment)
#944 (comment)

Also, see this updated sample.

[sobychacko]

I am going to close this issue through some documentation updates. Feel free to re-open if you are still facing issues.