HADOOP-18535. Implement token storage solution based on MySQL

[hchaverri]

Description of PR

These changes follow the same logic used in the existing ZK based secret manager implementation.

1. Minor changes to DelegationTokenInformation class so it's easier to convert to byte array format.
2. SQLDelegationTokenSecretManager transforms tokens and delegation keys into expected format and delegates the storage operations to the implementation class.
3. SQLDelegationTokenSecretManagerImpl interacts with SQL storage.
4. DistributedSQLCounter is used to manage sequenceNumbers and delegationKeyIds in a distributed way.
5. Including SQL schema files to create sample database, tables and users.

How was this patch tested?

1. Executed operations for token creation, renewal and cancellation. Compared results with existing ZK implementation.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 38s		Maven dependency ordering for branch
+1 💚	mvninstall	26m 7s		trunk passed
+1 💚	compile	23m 21s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	20m 51s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 10s		trunk passed
+1 💚	mvnsite	4m 33s		trunk passed
+1 💚	javadoc	3m 36s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 53s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	1m 22s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	22m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 39s		Maven dependency ordering for patch
+1 💚	mvninstall	2m 4s		the patch passed
+1 💚	compile	22m 44s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	22m 44s		the patch passed
+1 💚	compile	20m 53s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	20m 53s		the patch passed
+1 💚	blanks	0m 1s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 6s	/results-checkstyle-root.txt	root: The patch generated 7 new + 24 unchanged - 0 fixed = 31 total (was 24)
+1 💚	mvnsite	4m 21s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
-1 ❌	javadoc	1m 18s	/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt	hadoop-common-project_hadoop-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	javadoc	3m 46s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	0m 50s		hadoop-project has no data from spotbugs
-1 ❌	spotbugs	1m 58s	/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html	hadoop-hdfs-project/hadoop-hdfs-rbf generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	shadedclient	22m 32s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 49s		hadoop-project in the patch passed.
+1 💚	unit	19m 26s		hadoop-common in the patch passed.
+1 💚	unit	22m 52s		hadoop-hdfs-rbf in the patch passed.
-1 ❌	asflicense	1m 25s	/results-asflicense.txt	The patch generated 1 ASF License warnings.
		270m 59s

Reason	Tests
SpotBugs	module:hadoop-hdfs-project/hadoop-hdfs-rbf
	org.apache.hadoop.hdfs.server.federation.router.security.token.DistributedSQLCounter.selectCounterValue(boolean, Connection) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at DistributedSQLCounter.java:clean up java.sql.ResultSet Obligation to clean up resource created at DistributedSQLCounter.java:[line 66] is not discharged
	org.apache.hadoop.hdfs.server.federation.router.security.token.SQLDelegationTokenSecretManagerImpl.selectDelegationKey(int) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:[line 172] is not discharged
	org.apache.hadoop.hdfs.server.federation.router.security.token.SQLDelegationTokenSecretManagerImpl.selectTokenInfo(int, byte[]) may fail to clean up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:clean up java.sql.ResultSet Obligation to clean up resource created at SQLDelegationTokenSecretManagerImpl.java:[line 126] is not discharged

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/artifact/out/Dockerfile
GITHUB PR	#5159
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 7d8a8ec565f2 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 54419bb
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/testReport/
Max. process+thread count	2398 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 8s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	16m 35s		Maven dependency ordering for branch
+1 💚	mvninstall	29m 24s		trunk passed
+1 💚	compile	25m 51s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	22m 11s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	4m 31s		trunk passed
+1 💚	mvnsite	3m 48s		trunk passed
+1 💚	javadoc	3m 13s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 3s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	1m 0s		branch/hadoop-project no spotbugs output file (spotbugsXml.xml)
+1 💚	shadedclient	24m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 59s		the patch passed
+1 💚	compile	24m 49s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	24m 49s		the patch passed
+1 💚	compile	22m 10s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	22m 10s		the patch passed
-1 ❌	blanks	0m 0s	/blanks-eol.txt	The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
-0 ⚠️	checkstyle	4m 20s	/results-checkstyle-root.txt	root: The patch generated 7 new + 24 unchanged - 0 fixed = 31 total (was 24)
+1 💚	mvnsite	3m 48s		the patch passed
+1 💚	shellcheck	0m 1s		No new issues.
+1 💚	javadoc	3m 5s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	3m 2s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 🆗	spotbugs	0m 44s		hadoop-project has no data from spotbugs
+1 💚	shadedclient	24m 50s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 43s		hadoop-project in the patch passed.
+1 💚	unit	19m 5s		hadoop-common in the patch passed.
+1 💚	unit	34m 12s		hadoop-hdfs-rbf in the patch passed.
+1 💚	asflicense	1m 12s		The patch does not generate ASF License warnings.
		292m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/artifact/out/Dockerfile
GITHUB PR	#5159
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint shellcheck shelldocs
uname	Linux aeb70bb10131 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 0cd73b1
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/testReport/
Max. process+thread count	3144 (vs. ulimit of 5500)
modules	C: hadoop-project hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-rbf U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5159/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.