HADOOP-18510. Support confidential client in Azure refresh token gran… #5082

qcastel · 2022-10-27T08:30:15Z

Description of PR

Enforcing public client is a bad idea. We should not weaker our software just so the hadoop connector is able to refresh the token.
Therefore the idea would be to support both public and confidential OAuth2 client.

The fix is pretty straight forward, it consist of transiting the client secret to the azure refresh token grant flow and inject it into the request if present.

How was this patch tested?

Not yet, will see how I can test it quickly on our side with a patch.

For code changes:

[ x ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?

…t flow

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java

hadoop-yetus · 2022-10-27T10:16:58Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 54s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 43s		trunk passed
+1 💚	compile	0m 47s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 41s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 49s		trunk passed
+1 💚	javadoc	0m 46s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 36s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 21s		trunk passed
+1 💚	shadedclient	23m 55s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 37s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 30s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 21s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
-1 ❌	javadoc	0m 27s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt	hadoop-tools_hadoop-azure-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
-1 ❌	javadoc	0m 25s	/results-javadoc-javadoc-hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt	hadoop-tools_hadoop-azure-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~~20.04-b07 with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~~20.04-b07 generated 1 new + 15 unchanged - 0 fixed = 16 total (was 15)
+1 💚	spotbugs	1m 8s		the patch passed
+1 💚	shadedclient	23m 43s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 3s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 39s		The patch does not generate ASF License warnings.
		105m 21s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5082/1/artifact/out/Dockerfile
GITHUB PR	#5082
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 28723c93ba86 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b27aecd`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5082/1/testReport/
Max. process+thread count	533 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5082/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

ashutoshcipher

Thanks @qcastel for your contribution. Can you please add UT if possible and also run Azure integrations tests and add as part of this PR?

Ref for Integration tests - https://hadoop.apache.org/docs/stable/hadoop-azure/testing_azure.html#Testing_the_hadoop-azure_Module

qcastel · 2022-10-28T10:03:47Z

@ashutoshcipher I had a look at creating UT and it seems that there isn't any test yet covering the class impacted by this fix.

Mocking is also made difficult by the HTTP request made behind. Perhaps you got a HTTP mocking framework already used in this repo that I could use? Hoverfly for example?

Concerning the IT, looking at the XML to connect to the blob storage, they all uses account access keys. As the fix is for connecting with OAuth2, that won't be cover by the IT for sure.

Kind of feel that currently, all your coverage is missing testing the OAuth2 layer of Azure.

Would someone of your team could first write a test that cover the existing code? This way adding a test in this PR should be quite straight forward.

steveloughran · 2022-10-28T10:28:28Z

@snvijaya

hadoop-yetus · 2022-10-28T13:05:43Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 9s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	45m 3s		trunk passed
+1 💚	compile	0m 56s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 58s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	checkstyle	0m 48s		trunk passed
+1 💚	mvnsite	0m 59s		trunk passed
+1 💚	javadoc	0m 58s		trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 32s		trunk passed
+1 💚	shadedclient	27m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 41s		the patch passed
+1 💚	compile	0m 47s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 47s		the patch passed
+1 💚	compile	0m 41s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	javac	0m 41s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 27s		the patch passed
+1 💚	mvnsite	0m 43s		the patch passed
+1 💚	javadoc	0m 35s		the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 💚	spotbugs	1m 20s		the patch passed
+1 💚	shadedclient	26m 54s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	2m 18s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 45s		The patch does not generate ASF License warnings.
		117m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5082/2/artifact/out/Dockerfile
GITHUB PR	#5082
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux f6d83f6e203b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a392ed1`
Default Java	Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5082/2/testReport/
Max. process+thread count	539 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5082/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2022-10-31T19:30:49Z

going to invite @snvijaya to look at this.

qcastel · 2023-01-03T08:01:13Z

Any updates on your side @snvijaya ?

qcastel · 2023-09-19T13:53:02Z

Soon it will be the anniversary of this PR
@snvijaya @ashutoshcipher should we do a money pot?

HADOOP-18510. Support confidential client in Azure refresh token gran…

b27aecd

…t flow

qcastel commented Oct 27, 2022

View reviewed changes

...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java Show resolved Hide resolved

ashutoshcipher suggested changes Oct 27, 2022

View reviewed changes

Fix javadoc and make the client id option in refresh token flow

a392ed1

qcastel requested review from ashutoshcipher and steveloughran and removed request for ashutoshcipher and steveloughran October 28, 2022 11:06

qcastel closed this Dec 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-18510. Support confidential client in Azure refresh token gran… #5082

HADOOP-18510. Support confidential client in Azure refresh token gran… #5082

qcastel commented Oct 27, 2022

hadoop-yetus commented Oct 27, 2022

ashutoshcipher left a comment

qcastel commented Oct 28, 2022

steveloughran commented Oct 28, 2022

hadoop-yetus commented Oct 28, 2022

steveloughran commented Oct 31, 2022

qcastel commented Jan 3, 2023

qcastel commented Sep 19, 2023

HADOOP-18510. Support confidential client in Azure refresh token gran… #5082

HADOOP-18510. Support confidential client in Azure refresh token gran… #5082

Conversation

qcastel commented Oct 27, 2022

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Oct 27, 2022

ashutoshcipher left a comment

Choose a reason for hiding this comment

qcastel commented Oct 28, 2022

steveloughran commented Oct 28, 2022

hadoop-yetus commented Oct 28, 2022

steveloughran commented Oct 31, 2022

qcastel commented Jan 3, 2023

qcastel commented Sep 19, 2023