HADOOP-18610: [ABFS] OAuth2 Token Provider support for Azure Workload Identity #6787

anujmodi2021 · 2024-05-01T05:57:32Z

Description of PR

Jira: https://issues.apache.org/jira/browse/HADOOP-18610
Code Ported from PR: #5953

Please refer to the original PR Description below. This PR take forward the work done by original author @creste.

Resolved comments on original PR.
Some PR Checks were taken care of.
No new test added, nor any core functionality changed.

Original PR Description

Add support for Azure Active Directory (Azure AD) workload identities which integrate with the Kubernetes's native capabilities to federate with any external identity provider.

This PR is based on Haifeng Chen's patch attached to HADOOP-18610. I fixed a few typos and linter errors but did not modify the core functionality.

How was this patch tested?

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth

[WARNING] Tests run: 142, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 617, Failures: 0, Errors: 0, Skipped: 73
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 54

============================================================
HNS-SharedKey

[WARNING] Tests run: 142, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 617, Failures: 0, Errors: 0, Skipped: 28
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 41

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 142, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 601, Failures: 0, Errors: 0, Skipped: 268
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 44

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 142, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 617, Failures: 0, Errors: 0, Skipped: 75
[WARNING] Tests run: 380, Failures: 0, Errors: 0, Skipped: 78

Time taken: 57 mins 20 secs.

…Identity

…OP-18610

...ls/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java

hadoop-yetus · 2024-05-01T08:24:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+0 🆗	yamllint	0m 1s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 35s		trunk passed
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 5s		trunk passed
+1 💚	shadedclient	39m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 19s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 4s		the patch passed
+1 💚	shadedclient	39m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 26s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		145m 43s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/1/artifact/out/Dockerfile
GITHUB PR	#6787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint yamllint
uname	Linux 5bb3c7d57024 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `20a660a`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/1/testReport/
Max. process+thread count	528 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

snvijaya

+1

anujmodi2021 · 2024-05-06T05:41:37Z

Hi @steveloughran @mukund-thakur
Kindly requesting you to review this PR.

mukund-thakur

Added few comments:

...-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/WorkloadIdentityTokenProvider.java

mukund-thakur · 2024-05-06T20:27:45Z

...-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/WorkloadIdentityTokenProvider.java

+    // In case token is not refreshed for 1 hr or any clock skew issues,
+    // refresh token.
+    expiring = elapsedTimeSinceLastTokenRefreshInMillis >= ONE_HOUR
+        || elapsedTimeSinceLastTokenRefreshInMillis < 0;


So in case of clock skew we will be renewing during every call?
I would at least add a warning message.

There is a debug log added. Should I change it to warn then??

This is also same logic ported from MSI Token provider.

Retaining the logic of refreshing tokens in case of clock skews..

hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md

hadoop-yetus · 2024-05-08T10:29:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
			_ Prechecks _
+1 💚	dupname	0m 02s		No case conflicting files found.
+0 🆗	spotbugs	0m 00s		spotbugs executables are not available.
+0 🆗	codespell	0m 00s		codespell was not available.
+0 🆗	detsecrets	0m 00s		detect-secrets was not available.
+0 🆗	markdownlint	0m 00s		markdownlint was not available.
+0 🆗	yamllint	0m 00s		yamllint was not available.
+1 💚	@author	0m 01s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 00s		The patch appears to include 4 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	86m 54s		trunk passed
+1 💚	compile	4m 47s		trunk passed
+1 💚	checkstyle	4m 33s		trunk passed
+1 💚	mvnsite	4m 54s		trunk passed
+1 💚	javadoc	4m 33s		trunk passed
+1 💚	shadedclient	144m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 40s		the patch passed
+1 💚	compile	2m 09s		the patch passed
+1 💚	javac	2m 09s		the patch passed
+1 💚	blanks	0m 00s		The patch has no blanks issues.
+1 💚	checkstyle	2m 04s		the patch passed
+1 💚	mvnsite	2m 27s		the patch passed
+1 💚	javadoc	2m 03s		the patch passed
+1 💚	shadedclient	149m 24s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	asflicense	5m 15s		The patch does not generate ASF License warnings.
		403m 30s

Subsystem	Report/Notes
GITHUB PR	#6787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint yamllint
uname	MINGW64_NT-10.0-17763 710935166a77 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys
Build tool	maven
Personality	/c/hadoop/dev-support/bin/hadoop.sh
git revision	trunk / `20a660a`
Default Java	Azul Systems, Inc.-1.8.0_332-b09
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6787/1/testReport/
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6787/1/console
versions	git=2.44.0.windows.1
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java

hadoop-yetus · 2024-05-31T12:58:44Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 5 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 10s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 29s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 32s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 7s		trunk passed
+1 💚	shadedclient	39m 0s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 3 new + 4 unchanged - 0 fixed = 7 total (was 4)
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 5s		the patch passed
+1 💚	shadedclient	38m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		145m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/2/artifact/out/Dockerfile
GITHUB PR	#6787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint yamllint
uname	Linux f2b5609b471d 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `9a4a15e`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/2/testReport/
Max. process+thread count	528 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

anujmodi2021 · 2024-06-03T04:22:17Z

@rakeshadr @mukund-thakur
Thanks for the review. I have addressed all your comments.
Let me know if this looks good.

Thanks.

hadoop-yetus · 2024-06-03T07:09:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	18m 21s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	52m 58s		trunk passed
+1 💚	compile	0m 41s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	compile	0m 37s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 42s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 6s		trunk passed
+1 💚	shadedclient	39m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 29s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	javac	0m 29s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 19s	/results-checkstyle-hadoop-tools_hadoop-azure.txt	hadoop-tools/hadoop-azure: The patch generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4)
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
+1 💚	spotbugs	1m 5s		the patch passed
+1 💚	shadedclient	38m 58s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 30s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		167m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/3/artifact/out/Dockerfile
GITHUB PR	#6787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint yamllint
uname	Linux 43b360f269b0 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `962f86a`
Default Java	Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/3/testReport/
Max. process+thread count	532 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur

LGTM pending one minor comment

mukund-thakur · 2024-06-03T21:32:02Z

...-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/WorkloadIdentityTokenProvider.java

-    expiring = elapsedTimeSinceLastTokenRefreshInMillis >= ONE_HOUR
-        || elapsedTimeSinceLastTokenRefreshInMillis < 0;
+        System.currentTimeMillis() - tokenFetchTime;
+    boolean expiring = elapsedTimeSinceLastTokenRefreshInMillis < 0;


do we need this expiring variable now?

Not necessarily. It is just there for a cleaner code.
If we get rid of it, we will have this replaced with elapsedTimeSinceLastTokenRefreshInMillis < 0 at two places.

Seems okay to me that way as well.
Will take this,

ah..sorry somehow I missed that expiring is getting returned. I think it was better before.
Do you mind reverting this part?
so sorry for the rework.

Reverted...

hadoop-yetus · 2024-06-07T07:40:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	18m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	51m 24s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	0m 35s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	1m 6s		trunk passed
+1 💚	shadedclient	39m 49s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 30s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 26s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 25s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	1m 5s		the patch passed
+1 💚	shadedclient	38m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 29s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		162m 2s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/4/artifact/out/Dockerfile
GITHUB PR	#6787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint yamllint
uname	Linux 8f8f170ec433 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `af69bf2`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/4/testReport/
Max. process+thread count	527 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

phillipgibson · 2024-06-11T01:13:07Z

Do we have any blockers to get this merged?

anujmodi2021 · 2024-06-11T04:04:28Z

Do we have any blockers to get this merged?

Seems like no more blockers...

@mukund-thakur if this looks good, requesting you to please merge this. This is urgently required for a couple of high priority scenarios...

Thanks

hadoop-yetus · 2024-06-11T06:57:37Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	32m 15s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	markdownlint	0m 0s		markdownlint was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 6 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 45s		trunk passed
+1 💚	compile	0m 39s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	0m 36s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	0m 31s		trunk passed
+1 💚	mvnsite	0m 39s		trunk passed
+1 💚	javadoc	0m 38s		trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 34s		trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	1m 7s		trunk passed
+1 💚	shadedclient	39m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	0m 31s		the patch passed
+1 💚	compile	0m 27s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	0m 27s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 20s		the patch passed
+1 💚	mvnsite	0m 31s		the patch passed
+1 💚	javadoc	0m 27s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	0m 24s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	1m 4s		the patch passed
+1 💚	shadedclient	38m 51s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 34s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		174m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/5/artifact/out/Dockerfile
GITHUB PR	#6787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint yamllint
uname	Linux 6cf7c6a0fbfc 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17 09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `eb2b402`
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/5/testReport/
Max. process+thread count	534 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6787/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

mukund-thakur

LGTM +1

mukund-thakur · 2024-06-11T18:07:28Z

@anujmodi2021 could you please create a backport pr on top of branch-3.4 and re tetst and then we can merge that as well? Thanks

… Identity (apache#6787) Add support for Azure Active Directory (Azure AD) workload identities which integrate with the Kubernetes's native capabilities to federate with any external identity provider. Contributed By: Anuj Modi

creste and others added 16 commits August 16, 2023 09:39

HADOOP-18610 - ABFS OAuth2 Token Provider support for Azure Workload …

6629f9c

…Identity

Add license.

0a649d5

Remove whitespace.

2632811

Use string constants.

088b5c3

Add braces.

d47fd76

Explicitly specify POST HTTP method.

7d4cdb3

Make getClientAssertion() non-static.

524ee18

Add unit tests.

08da2ea

Remove unused import.

06fa885

Add package-info.java.

fe75cd3

Fix package name.

81c1b62

Add citation for token refresh interval.

3aeacfa

Add more unit tests. Add Azure environment variables to documentation.

329e3c9

Reduce line length.

f8be1ca

Merge branch 'workloadidentity' of github.com:creste/hadoop into HADO…

781b831

…OP-18610

PR Checks and Comments

20a660a

github-actions bot added trunk TOOLS ABFS labels May 1, 2024

anujmodi2021 marked this pull request as ready for review May 1, 2024 07:03

anujmodi2021 mentioned this pull request May 1, 2024

HADOOP-18610. ABFS OAuth2 Token Provider support for Azure Workload Identity #5953

Closed

4 tasks

snvijaya suggested changes May 1, 2024

View reviewed changes

...ls/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/AbstractAbfsIntegrationTest.java Show resolved Hide resolved

snvijaya approved these changes May 1, 2024

View reviewed changes

Sebasdevrs approved these changes May 2, 2024

View reviewed changes

mukund-thakur requested changes May 6, 2024

View reviewed changes

rakeshadr reviewed May 23, 2024

View reviewed changes

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java Outdated Show resolved Hide resolved

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java Show resolved Hide resolved

anujmodi2021 added 2 commits May 30, 2024 22:15

Merge branch 'trunk' into HADOOP-18610

7a2c74d

Resolved Comments

9a4a15e

Fixing test

962f86a

mukund-thakur reviewed Jun 3, 2024

View reviewed changes

Resolved Comments and Checkstyle Issues

af69bf2

Addressing Comments

eb2b402

mukund-thakur approved these changes Jun 11, 2024

View reviewed changes

mukund-thakur merged commit 005030f into apache:trunk Jun 11, 2024
4 checks passed

anujmodi2021 mentioned this pull request Jun 12, 2024

HADOOP-18610: [ABFS] [Backport to 3.4] OAuth2 Token Provider support for Azure Workload Identity #6881

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18610: [ABFS] OAuth2 Token Provider support for Azure Workload Identity #6787

HADOOP-18610: [ABFS] OAuth2 Token Provider support for Azure Workload Identity #6787

anujmodi2021 commented May 1, 2024 •

edited

Loading

hadoop-yetus commented May 1, 2024

snvijaya left a comment

anujmodi2021 commented May 6, 2024

mukund-thakur left a comment

mukund-thakur May 6, 2024

anujmodi2021 May 9, 2024

anujmodi2021 May 9, 2024

anujmodi2021 May 31, 2024

hadoop-yetus commented May 8, 2024

hadoop-yetus commented May 31, 2024

anujmodi2021 commented Jun 3, 2024

hadoop-yetus commented Jun 3, 2024

mukund-thakur left a comment

mukund-thakur Jun 3, 2024

anujmodi2021 Jun 7, 2024 •

edited

Loading

mukund-thakur Jun 10, 2024

anujmodi2021 Jun 11, 2024

hadoop-yetus commented Jun 7, 2024

phillipgibson commented Jun 11, 2024

anujmodi2021 commented Jun 11, 2024

hadoop-yetus commented Jun 11, 2024

mukund-thakur left a comment

mukund-thakur commented Jun 11, 2024

HADOOP-18610: [ABFS] OAuth2 Token Provider support for Azure Workload Identity #6787

HADOOP-18610: [ABFS] OAuth2 Token Provider support for Azure Workload Identity #6787

Conversation

anujmodi2021 commented May 1, 2024 • edited Loading

Description of PR

Original PR Description

How was this patch tested?

============================================================ HNS-OAuth

============================================================ HNS-SharedKey

============================================================ NonHNS-SharedKey

============================================================ AppendBlob-HNS-OAuth

hadoop-yetus commented May 1, 2024

snvijaya left a comment

Choose a reason for hiding this comment

anujmodi2021 commented May 6, 2024

mukund-thakur left a comment

Choose a reason for hiding this comment

mukund-thakur May 6, 2024

Choose a reason for hiding this comment

anujmodi2021 May 9, 2024

Choose a reason for hiding this comment

anujmodi2021 May 9, 2024

Choose a reason for hiding this comment

anujmodi2021 May 31, 2024

Choose a reason for hiding this comment

hadoop-yetus commented May 8, 2024

hadoop-yetus commented May 31, 2024

anujmodi2021 commented Jun 3, 2024

hadoop-yetus commented Jun 3, 2024

mukund-thakur left a comment

Choose a reason for hiding this comment

mukund-thakur Jun 3, 2024

Choose a reason for hiding this comment

anujmodi2021 Jun 7, 2024 • edited Loading

Choose a reason for hiding this comment

mukund-thakur Jun 10, 2024

Choose a reason for hiding this comment

anujmodi2021 Jun 11, 2024

Choose a reason for hiding this comment

hadoop-yetus commented Jun 7, 2024

phillipgibson commented Jun 11, 2024

anujmodi2021 commented Jun 11, 2024

hadoop-yetus commented Jun 11, 2024

mukund-thakur left a comment

Choose a reason for hiding this comment

mukund-thakur commented Jun 11, 2024

anujmodi2021 commented May 1, 2024 •

edited

Loading

============================================================
HNS-OAuth

============================================================
HNS-SharedKey

============================================================
NonHNS-SharedKey

============================================================
AppendBlob-HNS-OAuth

anujmodi2021 Jun 7, 2024 •

edited

Loading