HADOOP-17725. Keep MSI tenant ID and client ID optional (ADDENDUM)

[virajjasani]

Description of PR

Keep MSI tenant ID and client ID optional.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 52s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 30s		trunk passed
+1 💚	compile	0m 37s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	0m 31s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	checkstyle	0m 24s		trunk passed
+1 💚	mvnsite	0m 37s		trunk passed
+1 💚	javadoc	0m 29s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 28s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 1s		trunk passed
+1 💚	shadedclient	23m 6s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 29s		the patch passed
+1 💚	compile	0m 30s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	0m 30s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 16s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 22s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javadoc	0m 20s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	spotbugs	1m 3s		the patch passed
+1 💚	shadedclient	22m 56s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 58s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 29s		The patch does not generate ASF License warnings.
		93m 59s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/artifact/out/Dockerfile
GITHUB PR	#3788
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell
uname	Linux f282f7126c43 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 207820e
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/testReport/
Max. process+thread count	523 (vs. ulimit of 5500)
modules	C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3788/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

[virajjasani]

Storage account: US-West

$ mvn test

[INFO] Results:
[INFO] 
[WARNING] Tests run: 356, Failures: 0, Errors: 0, Skipped: 107
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:23 min
[INFO] Finished at: 2021-12-12T01:43:57+05:30
[INFO] ------------------------------------------------------------------------

HNS-Sharedkey:

$ ./dev-support/testrun-scripts/runtests.sh -c "HNS-SharedKey"

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  9.953 s
[INFO] Finished at: 2021-12-12T01:30:00+05:30
[INFO] ------------------------------------------------------------------------

Running the combination: HNS-SharedKey...
----- Test results -----
[INFO] Results:
[INFO] 
[WARNING] Tests run: 106, Failures: 0, Errors: 0, Skipped: 26
[INFO] Results:
[INFO] 
[WARNING] Tests run: 566, Failures: 0, Errors: 0, Skipped: 566
[INFO] Results:
[INFO] 
[WARNING] Tests run: 263, Failures: 0, Errors: 0, Skipped: 263

Time taken: 1 mins 4 secs.

However, integration tests are failing with StorageException: The account being accessed does not support http even though I have enableHttpsTrafficOnly as true.

[virajjasani]

Tried setting OAuth with Msi Token provider but looks like I am missing some config.

If I use these configs,

    <property>
        <name>fs.azure.account.auth.type.abfsuswest1</name>
        <value>OAuth</value>
        <description>The authorization type can be SharedKey, OAuth, Custom or SAS. The
            default is SharedKey.</description>
    </property>

    <property>
      <name>fs.azure.account.oauth.provider.type.abfsuswest1</name>
      <value>org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider</value>
      <description>The full name of token provider class name.</description>
    </property>

    <property>
        <name>fs.azure.account.oauth2.msi.tenant.abfsuswest1</name>
        <value>${value}</value>
        <description>msi tenantGuid.</description>
    </property>

    <property>
        <name>fs.azure.account.oauth2.client.id.abfsuswest1</name>
        <value>${value}</value>
        <description>AAD client id.</description>
    </property>

I get this error:

  <testcase name="test" classname="org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider" time="0">
    <skipped message="Not set: fs.azure.account.oauth.provider.type"/>
  </testcase>

However, adding fs.azure.account.oauth.provider.type with value org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider results in error:

[INFO] Running org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.382 s <<< FAILURE! - in org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider
[ERROR] test(org.apache.hadoop.fs.azurebfs.ITestAbfsMsiTokenProvider)  Time elapsed: 0.309 s  <<< ERROR!
java.lang.IllegalArgumentException: Illegal character in authority at index 8: abfss://abfs-testcontainer-db25b542-c2c9-4a07-b1da-e69d40f61d36@{account name}.dfs.core.windows.net
	at java.net.URI.create(URI.java:852)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:155)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3459)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:171)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3564)
	at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3517)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:591)
	at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:603)
	at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.createFileSystem(AbstractAbfsIntegrationTest.java:303)
	at org.apache.hadoop.fs.azurebfs.AbstractAbfsIntegrationTest.setup(AbstractAbfsIntegrationTest.java:174)

Is this a known issue with some missing config while running test?

mvn command:

$ mvn -T 1C clean verify -Dtest=none -Dit.test=ITestAbfsMsiTokenProvider

[anmolanmol1234]

For MSITokenProvider, there are 4 mandatory fields that need to be provided - The clientid, authEndPoint, tenantGuid and authority, The authEndPoint and authority configs are missing here. Can you please try once by adding these config values as well ?