Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bad SASL negotiation status: 3 (GSS initiate failed) #172

Open
pksilen opened this issue Jan 25, 2022 · 12 comments
Open

Bad SASL negotiation status: 3 (GSS initiate failed) #172

pksilen opened this issue Jan 25, 2022 · 12 comments

Comments

@pksilen
Copy link

pksilen commented Jan 25, 2022

We experience an error when connecting to hive using Kerberos auth:
Bad SASL negotiation status: 3 (GSS initiate failed)

Below is our code and before executing our code, we execute kinit:

kinit -kt {{ .Values.krb5.keytabFile }} {{ .Values.krb5.principal }};


	configuration.Username = cfg.HiveUsername
	configuration.Password = cfg.HivePassword
	configuration.Service = cfg.HiveService
	configuration.FetchSize = cfg.HiveFetchsize

	if cfg.HiveAuth == "KERBEROS" || cfg.HiveAuth == "kerberos" {
		configuration.TLSConfig = &tls.Config{
			InsecureSkipVerify: true,
		}
	}

	connection, errConn := gohive.Connect(cfg.HiveHost, cfg.HivePort, cfg.HiveAuth, configuration)
	if errConn != nil {
		return nil, fmt.Errorf("Could not connect to Hive. %v", errConn)
	}

	return &HiveClient{
		Configuration: configuration,
		Connection:    connection,
	}, nil
@pksilen
Copy link
Author

pksilen commented Feb 10, 2022

Hi,
Any chance to someone look at this problem? We earlier had the same code functioning ok. We are using gohive version 1.4.0. No in two environments we get this same issue. Can this be a configuration issue?

@beltran
Copy link
Owner

beltran commented Feb 10, 2022 •
edited
Loading

Hello, I missed this.

From the code you're using ssl and kerberos right? This should be working, you can see this test as an example of how to set this. My guess is that this is a configuration issue, seems like something is wrong with kerberos, it maybe the krb5.conf file. This is the one used for the tests. Also please check how the tests do kinit, which may solve your problem.

After you do kinit ..., what does klist display? Also useful would be the hive logs when you try to connect, there should be a stacktrace.

@beltran
Copy link
Owner

beltran commented Feb 25, 2022

Hello, were you finally able to resolve this?

@pksilen
Copy link
Author

pksilen commented Feb 25, 2022

It is not yet solved but is being investigated by our data lake expert. We found "Checksum failed" error in Hive logs:

ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failure2022-02-14 16:35:45,471 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) ~[?:1.8.0_292] at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:537) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292] at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_292] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) [hadoop-common-3.0.0-cdh6.3.2.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:858) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: sun.security.krb5.KrbCryptoException: Checksum failed at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: java.security.GeneralSecurityException: Checksum failed at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 more

@KeepFire8916
Copy link

Hello, did you finally resolve this ? I've got the same issue like this

@adslen
Copy link

adslen commented Dec 22, 2023

The author has not handled the err here, and the actual errors are being concealed.
image

@beltran
Copy link
Owner

beltran commented Dec 22, 2023 •
edited
Loading

That is on purpose @adslen, the error may be set even when the context has been initialized successfully. But I'm happy to accept improvements if you can think of any.

@adslen
Copy link

adslen commented Dec 25, 2023

We encountered a problem similar to this last time. We spent a considerable amount of time attempting to resolve it, only to discover that the 'err' wasn't handled here, and the actual error wasn't being thrown. After reviewing the code briefly, it seems that the situation you mentioned occurs only when the error is 'ErrContinueNeeded.' Perhaps we could handle this similarly to GORM's handling of 'gorm.ErrRecordNotFound,' where we throw the error and let the user decide whether to handle the exception. Like this:
image

@beltran
Copy link
Owner

beltran commented Dec 25, 2023

Sorry you had to spend so much time, and thank you for your suggestion. Is this the fix you are proposing? If not I would appreciate if you could create the pull request.

@Azusain
Copy link

Azusain commented May 30, 2024 •
edited
Loading

It is not yet solved but is being investigated by our data lake expert. We found "Checksum failed" error in Hive logs:

ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failure2022-02-14 16:35:45,471 ERROR org.apache.thrift.transport.TSaslTransport: [HiveServer2-Handler-Pool: Thread-47]: SASL negotiation failurejavax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) ~[?:1.8.0_292] at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:537) ~[hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:652) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_292] at javax.security.auth.Subject.doAs(Subject.java:360) [?:1.8.0_292] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1855) [hadoop-common-3.0.0-cdh6.3.2.jar:?] at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:649) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) [hive-exec-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed) at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:858) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: sun.security.krb5.KrbCryptoException: Checksum failed at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 moreCaused by: java.security.GeneralSecurityException: Checksum failed at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100) ~[?:1.8.0_292] at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94) ~[?:1.8.0_292] at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_292] at sun.security.krb5.KrbApReq.(KrbApReq.java:149) ~[?:1.8.0_292] at sun.security.jgss.krb5.InitSecContextToken.(InitSecContextToken.java:140) ~[?:1.8.0_292] at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:831) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_292] at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_292] at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:167) ~[?:1.8.0_292] ... 14 more

I've encountered the same issue and got the same log output from Hive server. How did you fix that?

@KeepFire8916
Copy link

KeepFire8916 commented Jun 11, 2024 via email

@Azusain
Copy link

Azusain commented Jun 25, 2024

I've already solved this problem and it was my foolish mistake that caused it: I tried to export .keytab file by using 'ktadd' command and did't realize that it would make the KDC regenerate a new .keytab file, which appearently conflicted with the original one.
So in the end I just copied the .keytab file from the keytab path set in the hive-site.xml to my client, and it worked fine...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@pksilen @beltran @adslen @Azusain @KeepFire8916