HIVE-26632: Update DelegationTokenSecretManager current key ID to pre…

[cnauroth]

…vent erroneous database updates.

What changes were proposed in this pull request?

Update DelegationTokenSecretManager current key ID to prevent erroneous database updates.

Why are the changes needed?

When multiple HiveMetaStore processes share a database for MASTER_KEYS, this bug can cause incorrect update attempts and increased database load.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Run multiple HiveMetaStore processes sharing the same database. To make it easier to expose the problem, downtune configuration properties related to master key rollover to artificially low values:

• hive.cluster.delegation.token.gc-interval
• hive.cluster.delegation.key.update-interval
• hive.cluster.delegation.token.max-lifetime

WIthout the patch, stack traces like this are visible in the logs:

2022-08-03T00:09:48,744 ERROR [Thread[Thread-9,5,main]] thrift.TokenStoreDelegationTokenSecretManager: ExpiredTokenRemover thread received unexpected exception. org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: NoSuchObjectException(message:No key found with keyId: 1)
org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: NoSuchObjectException(message:No key found with keyId: 1)
	at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:170) ~[hive-exec-2.3.7.jar:2.3.7]
	at org.apache.hadoop.hive.thrift.DBTokenStore.updateMasterKey(DBTokenStore.java:51) ~[hive-exec-2.3.7.jar:2.3.7]
	at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.rollMasterKeyExt(TokenStoreDelegationTokenSecretManager.java:269) ~[hive-exec-2.3.7.jar:2.3.7]
	at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager$ExpiredTokenRemover.run(TokenStoreDelegationTokenSecretManager.java:301) [hive-exec-2.3.7.jar:2.3.7]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: org.apache.hadoop.hive.metastore.api.NoSuchObjectException: No key found with keyId: 1
	at org.apache.hadoop.hive.metastore.ObjectStore.updateMasterKey(ObjectStore.java:7727) ~[hive-exec-2.3.7.jar:2.3.7]

After applying the patch, the problem goes away.

The exact timing makes it difficult to reproduce accurately in a unit test.

[cnauroth]

http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3673/1/tests

There are 2 CLI integration test failures that don't appear to be related to this patch.

[cnauroth]

@ayushtkn or @dengzhhu653 , would one of you be able to review this? Thank you.

[ayushtkn]

LGTM. Thanx for fixing this!!

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
No Duplication information