[Bug] Connecting to Hive metastore using Kerberos

[shubham-goel99]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

I have encountered a bug in the Kyuubi server that is causing an undesired behavior. The Kyuubi server is kerberized, meaning that kerberos authentication is required when connecting to it, and as a result, there is a Service Principal Name (SPN) associated with it. The Kyuubi server is started by a user named 'A' and it has a Ticket-Granting Ticket (TGT) cache associated with it. Additionally, the Hive metastore is also kerberized, requiring the spark driver process to communicate with it using Kerberos authentication exclusively.

The issue arises when the spark driver process is spawned/started by the Kyuubi server with the same properties/environment variable (KRB5CCNAME) as the Kyuubi server itself. Since the Kyuubi server periodically runs kinit, the TGT cache gets overwritten with the SPN principal name. As a result, the spark driver process ends up using the SPN principal to connect to the Hive metastore instead of the desired behavior of using the user A principal. Finally, Kyuubi engine throws an error while connecting to Hive metastore saying that no user exist (as Hive metastore is expecting A's principal name but instead it gets SPN)

Here, Kyuubi acts both as a client and server and in these cases, there are two principals involved: one for the server side (acceptor - SPN) and one for when Kyuubi acts as a client (initiator). However, Kyuubi only supports one principal and forces its use on both sides.

Affects Version(s)

master

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

hadoop.security.authentication kerberos
kyuubi.authentication KERBEROS
kyuubi.kinit.principal HTTP/SPN@HOST.COM
kyuubi.kinit.keytab <keytab location>
hadoop.security.auth_to_local.mechanism MIT
kyuubi.authentication.sasl.qop auth-conf
kyuubi.frontend.ssl.keystore.password <SSL_PASSWORD>
kyuubi.frontend.ssl.keystore.path <KEYSTORE_PATH>
kyuubi.frontend.ssl.keystore.type JKS
kyuubi.frontend.thrift.binary.ssl.enabled true

Kyuubi Engine Configurations

spark.sql.warehouse.dir  <warehouse_dir_path>
spark.worker.cleanup.enabled True
spark.driver.defaultJavaOptions -Djava.security.debug=all -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.jgss.debug=true -Dsun.security.krb5.debug=true -Dsun.security.jgss.native=true -Dsun.security.jgss.lib=/usr/libexec/libgsswrap.so
spark.hadoop.javax.jdo.option.ConnectionURL jdbc:sqlserver://<server_name>;databaseName=<database_name>;authenticationScheme=JavaKerberos;trustStore=<truststore_path>;encrypt=false;integratedSecurity=true
spark.hadoop.javax.jdo.option.ConnectionDriverName com.microsoft.sqlserver.jdbc.SQLServerDriver
spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalogImplementation hive
spark.authenticate true
spark.authenticate.secret <temp_secret>
spark.network.crypto.enabled true
spark.io.encryption.enabled true

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @shubham-goel99,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[pan3793]

I don't think your concerns are actually existing. Kyuubi leverages the Hadoop Impersonation mechanism to support multi-tenants and simplify the keytab management.

PS: this reply may be too brief because Hadoop Security is really a big and complex topic, it may require writing a book to explain all the details. I suppose you have pieces of such domain knowledge, and I would like to supply more details based on your feedback.

[shubham-goel99]

My cluster is running in secure mode and user 'A' (user that started the Kyuubi server) doesn't have access to Kerberos credentials for any other user (even of a user that submitted the job to Kyuubi server) apart from its own user. Hence, I want to connect to hive metastore using user 'A' Kerberos credentials itself but it ends up picking SPN credentials.

Please note that I am not in the hadoop environment i.e not using HDFS nor Yarn.

[pan3793]

Your assumption skips the Hadoop Impersonation mechanism again.

The above Hadoop Impersonation mechanism also works with Spark on K8s smoothly.

Client (Kerberos auth with A)
  => Kyuubi Server (SPN who is configured to allow proxy any users)
    => Spark (use SPN's TGT but with `--proxy-user A`, to generate Delegation Tokens for user A then distribute it to Spark Driver/Executor)
      => Spark App runs with the Delegation Token for proxy user A

[mahalingamagesthian]

@shubham-goel99 -
Could you please try this ?
kyuubi.authentication=KERBEROS
kyuubi.kinit.principal=hive/FQDN@REALM.ORG
kyuubi.kinit.keytab=/path/to/keytab/hive.service.keytab

This will run the Kyuubi server process as hive user. It will perform the spark-submit on behalf of the user who has invoked it., as @pan3793 says: hive user will create the required delegation token on behalf of the user who has invoked it.
hive/FQDN@REALM.ORG - should be the SPN of HiveMetastore.
/path/to/keytab/hive.service.keytab - Keytab file of SPN of HiveMetastore