HDFS-15168: ABFS enhancement to translate AAD to Linux identities. #1978

amarnathkarthik · 2020-04-25T04:02:05Z

ABFS driver enhancement - Allow customizable translation from AAD SPNs and security groups to Linux user and group

Integration Test results - East US2:
Tests run: 1283, Failures: 0, Errors: 0, Skipped: 404

amarnathkarthik · 2020-04-25T04:07:25Z

@steveloughran I'd mess up with the previous PR-1858 while fixing the review comment which got auto closed. Created new PR as i want to follow the contributor git commit process. Please review.

CC: @virajith

hadoop-yetus · 2020-04-25T05:28:03Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 1s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 8s	trunk passed
+1 💚	compile	0m 32s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 35s	trunk passed
+1 💚	shadedclient	14m 59s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed
+0 🆗	spotbugs	0m 54s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 52s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s	the patch passed
+1 💚	compile	0m 23s	the patch passed
+1 💚	javac	0m 23s	the patch passed
-0 ⚠️	checkstyle	0m 16s	hadoop-tools/hadoop-azure: The patch generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
+1 💚	mvnsite	0m 28s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 48s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 22s	the patch passed
+1 💚	findbugs	0m 55s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 22s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		57m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/1/artifact/out/Dockerfile
GITHUB PR	#1978
JIRA Issue	HDFS-15168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 2a0849be385c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `453771f`
Default Java	Private Build-1.8.0_252-8u252-b09-1~16.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/1/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/1/testReport/
Max. process+thread count	413 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/1/console
versions	git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

steveloughran

looks like some of my previous comments have been addressed, but not all. go through them & makes sure they're all ticked off. These are all minor style/javadoc/import issues, so straightforward

...p-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformerInterface.java

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java

hadoop-yetus · 2020-04-28T01:27:36Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 31s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	18m 51s	trunk passed
+1 💚	compile	0m 31s	trunk passed
+1 💚	checkstyle	0m 25s	trunk passed
+1 💚	mvnsite	0m 35s	trunk passed
+1 💚	shadedclient	14m 52s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 27s	trunk passed
+0 🆗	spotbugs	0m 52s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 49s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 27s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
-0 ⚠️	checkstyle	0m 17s	hadoop-tools/hadoop-azure: The patch generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 43s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	the patch passed
+1 💚	findbugs	0m 54s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 22s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		56m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/2/artifact/out/Dockerfile
GITHUB PR	#1978
JIRA Issue	HDFS-15168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux f5bcc5a5f26e 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `5e0eda5`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/2/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/2/testReport/
Max. process+thread count	414 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/2/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

amarnathkarthik · 2020-04-28T04:21:45Z

@steveloughran Thanks on review this PR, addressed below review comments:

Trailing "."
Import ordering
Formatting of ternary operator
Changing to debug statement
Use of "#" as comment

steveloughran

production code LGTM, just some minor test changes, with the main one being that file creation. For the expected exceptions, if you want to leave as is --that's OK With me

...e/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestTextFileBasedIdentityHandler.java

hadoop-yetus · 2020-05-20T03:48:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s	The patch appears to include 1 new or modified test files.
		_ trunk Compile Tests _
+1 💚	mvninstall	19m 5s	trunk passed
+1 💚	compile	0m 31s	trunk passed
+1 💚	checkstyle	0m 24s	trunk passed
+1 💚	mvnsite	0m 34s	trunk passed
+1 💚	shadedclient	14m 35s	branch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 29s	trunk passed
+0 🆗	spotbugs	0m 53s	Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚	findbugs	0m 50s	trunk passed
		_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s	the patch passed
+1 💚	compile	0m 24s	the patch passed
+1 💚	javac	0m 24s	the patch passed
-0 ⚠️	checkstyle	0m 17s	hadoop-tools/hadoop-azure: The patch generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1)
+1 💚	mvnsite	0m 27s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	shadedclient	13m 31s	patch has no errors when building and testing our client artifacts.
+1 💚	javadoc	0m 24s	the patch passed
+1 💚	findbugs	0m 56s	the patch passed
		_ Other Tests _
+1 💚	unit	1m 25s	hadoop-azure in the patch passed.
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		56m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/4/artifact/out/Dockerfile
GITHUB PR	#1978
JIRA Issue	HDFS-15168
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname	Linux 9377aff02bd2 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	personality/hadoop.sh
git revision	trunk / `1983eea`
Default Java	Private Build-1.8.0_252-8u252-b09-1~18.04-b09
checkstyle	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/4/artifact/out/diff-checkstyle-hadoop-tools_hadoop-azure.txt
Test Results	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/4/testReport/
Max. process+thread count	454 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://builds.apache.org/job/hadoop-multibranch/job/PR-1978/4/console
versions	git=2.17.1 maven=3.6.0 findbugs=3.1.0-RC1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajith · 2020-05-29T02:00:17Z

Thanks for reviewing this @steveloughran! Merging this PR based on Steve's lgtm above!
Thanks for working on this @amarnathkarthik

…1978)

…pache#1978) (cherry picked from commit b2200a3)

…entities. (apache#1978) Conflicts: hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/AzureBlobFileSystemStore.java hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/constants/ConfigurationKeys.java Change-Id: I51afcf3ba01154925f3ec091962987cea71acdef

steveloughran requested changes Apr 27, 2020

View reviewed changes

...p-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/IdentityTransformerInterface.java Outdated Show resolved Hide resolved

...op-azure/src/main/java/org/apache/hadoop/fs/azurebfs/utils/TextFileBasedIdentityHandler.java Show resolved Hide resolved

amarnathkarthik requested a review from steveloughran May 4, 2020 18:26

steveloughran requested changes May 19, 2020

View reviewed changes

...e/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestTextFileBasedIdentityHandler.java Show resolved Hide resolved

...e/src/test/java/org/apache/hadoop/fs/azurebfs/services/TestTextFileBasedIdentityHandler.java Outdated Show resolved Hide resolved

HDFS-15168: ABFS enhancement to translate AAD to Linux identities.

c69c0fc

amarnathkarthik force-pushed the feature/HDFS-15168 branch from 8bc27d9 to c69c0fc Compare May 20, 2020 02:28

amarnathkarthik requested a review from steveloughran May 20, 2020 02:32

virajith merged commit b2200a3 into apache:trunk May 29, 2020

asfgit pushed a commit that referenced this pull request Jul 25, 2020

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. (#…

8b7e774

…1978)

arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 11, 2022

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. (a…

5d91b25

…pache#1978) (cherry picked from commit b2200a3)

arjun4084346 pushed a commit to arjun4084346/hadoop that referenced this pull request Apr 11, 2022

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. (a…

aa48774

…pache#1978) (cherry picked from commit b2200a3)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. #1978

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. #1978

amarnathkarthik commented Apr 25, 2020 •

edited

amarnathkarthik commented Apr 25, 2020

hadoop-yetus commented Apr 25, 2020

steveloughran left a comment

hadoop-yetus commented Apr 28, 2020

amarnathkarthik commented Apr 28, 2020

steveloughran left a comment

hadoop-yetus commented May 20, 2020

virajith commented May 29, 2020

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. #1978

HDFS-15168: ABFS enhancement to translate AAD to Linux identities. #1978

Conversation

amarnathkarthik commented Apr 25, 2020 • edited

amarnathkarthik commented Apr 25, 2020

hadoop-yetus commented Apr 25, 2020

steveloughran left a comment

Choose a reason for hiding this comment

hadoop-yetus commented Apr 28, 2020

amarnathkarthik commented Apr 28, 2020

steveloughran left a comment

Choose a reason for hiding this comment

hadoop-yetus commented May 20, 2020

virajith commented May 29, 2020

amarnathkarthik commented Apr 25, 2020 •

edited