YARN-11399 Make DelegationTokenRenwer timeout and retry feature configurable

[Daniel-009497]

ResourceManager Delegation token renew timeout and retry mechanism is introduced in Hadoop 3.3.1, The switch of this feature is already there, but it is true and can not be disabled.
So this patch introduce a configuration to make this feature configurable.

[slfan1989]

@Daniel-009497 Thank you very much for your contribution. From my personal point of view, this parameter should not be set to false. If it is set to false, RM will not be able to renew token. I think this change may be unreasonable.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 7s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	15m 39s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 56s		trunk passed
+1 💚	compile	11m 31s		trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	compile	9m 8s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	checkstyle	1m 52s		trunk passed
+1 💚	mvnsite	3m 4s		trunk passed
-1 ❌	javadoc	0m 53s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 22s		trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	6m 7s		trunk passed
+1 💚	shadedclient	25m 16s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 7s		the patch passed
+1 💚	compile	9m 53s		the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚	javac	9m 53s		the patch passed
+1 💚	compile	8m 51s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	javac	8m 51s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 39s		the patch passed
+1 💚	mvnsite	2m 45s		the patch passed
-1 ❌	javadoc	0m 51s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚	javadoc	2m 12s		the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚	spotbugs	6m 22s		the patch passed
+1 💚	shadedclient	25m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	1m 9s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt	hadoop-yarn-api in the patch passed.
+1 💚	unit	5m 34s		hadoop-yarn-common in the patch passed.
-1 ❌	unit	107m 0s	/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	1m 7s		The patch does not generate ASF License warnings.
		285m 42s

Reason	Tests
Failed junit tests	hadoop.yarn.conf.TestYarnConfigurationFields
	hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5234/1/artifact/out/Dockerfile
GITHUB PR	#5234
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
uname	Linux c8ca761653ab 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / f628abe
Default Java	Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5234/1/testReport/
Max. process+thread count	906 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5234/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[Daniel-009497]

@Daniel-009497 Thank you very much for your contribution. From my personal point of view, this parameter should not be set to false. If it is set to false, RM will not be able to renew token. I think this change may be unreasonable.

@slfan1989 Noy really, if the delegation token renewer pool tracker feature is disabled, the previous token renew logic still take effect. This new feature is just a enhancement feature based on currently exsiting token renew feature.
The reason why I wanna make it configurable is that we found several issues with this new feature.
Of course, we fixed the bugs found yet, but in case of more unknown ones, we'd better make the new feature configurable just in case. And it is a traditional way to make an alternaltive feature configurable in Hadoop community as always.

[slfan1989]

@Daniel-009497 I understand your explanation, it seems reasonable, I will check the current logic later.

hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer#testDTKeepAlive1/testDTKeepAlive2 seems to be affected, Please check if our changes cause this junit Test to fail.

[Daniel-009497]

@brahmareddybattula @ayushtkn @bibinchundatt @hemanthboyina

Could you pls help to review and merge?

[slfan1989]

@Daniel-009497 Thank you very much for your contribution, I read this part of code and I think this variable(delegationTokenRenewerPoolTrackerFlag) should always be true. We should not add configuration for this variable.

This optimization comes from JIRA: YARN-9768. RM Renew Delegation token thread should timeout and retry.
The description information explains the reason for the optimization:

This call is made to an underlying HDFS NN or Router Node (which has exact APIs as HDFS NN). 
If one of the nodes is bad and the renew call is stuck the thread remains stuck indefinitely. 
The thread should ideally timeout the renewToken and retry from the client's perspective.

If this variable is set to false, once the HDFS NN Or Router Node fails, the RM refresh thread will be stuck.

We can read YARN-9768 carefully. The author at that time should have set this variable for compatibility with previous unit tests. We can see the following code.

org.apache.hadoop.yarn.server.resourcemanager.security#DelegationTokenRenewer L939

 @VisibleForTesting
  public void setDelegationTokenRenewerPoolTracker(boolean flag) {
    delegationTokenRenewerPoolTrackerFlag = flag;
  }

Let's wait for other partners' suggestions.