Skip to content

HADOOP-17363. ABFS: Fix Oauth 2.0 username-password authentication issues #2445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
matsushita-shin wants to merge 1 commit into from
Closed

HADOOP-17363. ABFS: Fix Oauth 2.0 username-password authentication issues #2445

matsushita-shin wants to merge 1 commit into from
+74 −3

Conversation

@matsushita-shin
Copy link

@matsushita-shin matsushita-shin commented Nov 7, 2020

https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html

I have tried OAuth 2.0 authentication with the username and password written above.
However, it failed with the following exception.

Exception in thread "main" HTTP Error 400; url='https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token' AADToken: HTTP connection to https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token failed for getting token from AzureAD.; requestId='187c97a4-82a0-4b36-b764-XXXXXXXXXXXX'; contentType='application/json; charset=utf-8'; response '{"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'jiro' was not found in the directory '3070a5de-410e-4885-XXXX-XXXXXXXXXXXX'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 187c97a4-82a0-4b36-b764-a3b8b1c45201\r\nCorrelation ID: 4eb4a71e-2eef-4788-9c8c-24f4c84f6981\r\nTimestamp: 2020-11-07 11:49:21Z","error_codes":[700016],"timestamp":"2020-11-07 11:49:21Z","trace_id":"187c97a4-82a0-4b36-b764-a3b8b1c45201","correlation_id":"4eb4a71e-2eef-4788-9c8c-24f4c84f6981","error_uri":"https://login.microsoftonline.com/error?code=700016"}'org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error 400; url='https://login.microsoftonline.com/3070a5de-410e-4885-b6cd-95fe759ced2b/oauth2/token' AADToken: HTTP connection to https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token failed for getting token from AzureAD.; requestId='187c97a4-82a0-4b36-b764-XXXXXXXXXXXX'; contentType='application/json; charset=utf-8'; response '{"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'jiro' was not found in the directory '3070a5de-410e-4885-XXXX-XXXXXXXXXXXX'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 187c97a4-82a0-4b36-b764-a3b8b1c45201\r\nCorrelation ID: 4eb4a71e-2eef-4788-9c8c-24f4c84f6981\r\nTimestamp: 2020-11-07 11:49:21Z","error_codes":[700016],"timestamp":"2020-11-07 11:49:21Z","trace_id":"187c97a4-82a0-4b36-b764-a3b8b1c45201","correlation_id":"4eb4a71e-2eef-4788-9c8c-24f4c84f6981","error_uri":"https://login.microsoftonline.com/error?code=700016"}'
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:215)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:134)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.createPath(AbfsClient.java:293)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createDirectory(AzureBlobFileSystemStore.java:445)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.mkdirs(AzureBlobFileSystem.java:409)
	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2355)
	at com.sample.HelloWorld.main(HelloWorld.java:116)
Caused by: org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator$HttpException: HTTP Error 400; url='https://login.microsoftonline.com/3070a5de-410e-XXXX-XXXXXXXXXXXX/oauth2/token' AADToken: HTTP connection to https://login.microsoftonline.com/3070a5de-410e-4885-XXXX-XXXXXXXXXXXX/oauth2/token failed for getting token from AzureAD.; requestId='187c97a4-82a0-4b36-b764-a3b8b1c45201'; contentType='application/json; charset=utf-8'; response '{"error":"unauthorized_client","error_description":"AADSTS700016: Application with identifier 'jiro' was not found in the directory '3070a5de-410e-4885-XXXX-XXXXXXXXXXXX'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant.\r\nTrace ID: 187c97a4-82a0-4b36-b764-a3b8b1c45201\r\nCorrelation ID: 4eb4a71e-2eef-4788-9c8c-24f4c84f6981\r\nTimestamp: 2020-11-07 11:49:21Z","error_codes":[700016],"timestamp":"2020-11-07 11:49:21Z","trace_id":"187c97a4-82a0-4b36-b764-a3b8b1c45201","correlation_id":"4eb4a71e-2eef-4788-9c8c-24f4c84f6981","error_uri":"https://login.microsoftonline.com/error?code=700016"}'
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenSingleCall(AzureADAuthenticator.java:394)
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenCall(AzureADAuthenticator.java:291)
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenCall(AzureADAuthenticator.java:273)
	at org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenUsingClientCreds(AzureADAuthenticator.java:96)
	at org.apache.hadoop.fs.azurebfs.oauth2.UserPasswordTokenProvider.refreshToken(UserPasswordTokenProvider.java:54)
	at org.apache.hadoop.fs.azurebfs.oauth2.AccessTokenProvider.getToken(AccessTokenProvider.java:50)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.getAccessToken(AbfsClient.java:670)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.executeHttpOperation(AbfsRestOperation.java:168)
	... 6 more

The cause of the error seems to be that UserPasswordTokenProvider is calling getTokenUsingClientCreds() for the service principal.

https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth-ropc

I checked the API specifications of Azure and fixed the cause of this error.

@hadoop-yetus
Copy link

hadoop-yetus commented Nov 7, 2020

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 26m 48s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 markdownlint 0m 0s markdownlint was not available.
+1 💚 @author 0m 1s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 33m 47s trunk passed
+1 💚 compile 0m 37s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 compile 0m 35s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 checkstyle 0m 27s trunk passed
+1 💚 mvnsite 0m 38s trunk passed
+1 💚 shadedclient 16m 4s branch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 31s trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 0m 29s trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+0 🆗 spotbugs 0m 59s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 0m 56s trunk passed
_ Patch Compile Tests _
+1 💚 mvninstall 0m 30s the patch passed
+1 💚 compile 0m 30s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javac 0m 30s the patch passed
+1 💚 compile 0m 26s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 javac 0m 26s the patch passed
-0 ⚠️ checkstyle 0m 18s /diff-checkstyle-hadoop-tools_hadoop-azure.txt hadoop-tools/hadoop-azure: The patch generated 2 new + 2 unchanged - 0 fixed = 4 total (was 2)
+1 💚 mvnsite 0m 29s the patch passed
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 shadedclient 14m 27s patch has no errors when building and testing our client artifacts.
+1 💚 javadoc 0m 26s the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1
+1 💚 javadoc 0m 24s the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
+1 💚 findbugs 0m 57s the patch passed
_ Other Tests _
-1 ❌ unit 1m 28s /patch-unit-hadoop-tools_hadoop-azure.txt hadoop-azure in the patch passed.
+1 💚 asflicense 0m 33s The patch does not generate ASF License warnings.
103m 34s
Reason Tests
Failed junit tests hadoop.fs.azure.TestBlobMetadata
hadoop.fs.azure.TestNativeAzureFileSystemConcurrency
hadoop.fs.azure.TestNativeAzureFileSystemFileNameCheck
hadoop.fs.azure.TestWasbFsck
hadoop.fs.azure.TestNativeAzureFileSystemOperationsMocked
hadoop.fs.azure.TestOutOfBandAzureBlobOperations
hadoop.fs.azure.TestNativeAzureFileSystemContractMocked
hadoop.fs.azure.TestNativeAzureFileSystemMocked
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2445/1/artifact/out/Dockerfile
GITHUB PR #2445
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle markdownlint
uname Linux c8880252bfb0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / ae7b00a
Default Java Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2445/1/testReport/
Max. process+thread count 420 (vs. ulimit of 5500)
modules C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2445/1/console
versions git=2.17.1 maven=3.6.0 findbugs=4.1.3
Powered by Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

@steveloughran
Copy link
Contributor

steveloughran commented Nov 16, 2020

Those Test* failure are unrelated, but the testing abfs policy is always "PR supplier must declare which Azure endpoint they ran all the mvn verify integration tests against

Nominating @snvijaya and @bilaharith for review

mukund-thakur
mukund-thakur reviewed Jun 21, 2021
View reviewed changes
...ls/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/oauth2/AzureADAuthenticator.java
*/
public static AzureADToken getTokenUsingUserCreds(String authEndpoint,
String username, String password, String clientId,
String clientSecret) throws IOException {
Copy link
Contributor

@mukund-thakur mukund-thakur Jun 21, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like clientSecret is not required. Why are we still passing this?

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2025

We're closing this stale PR because it has been open for 100 days with no activity. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you feel like this was a mistake, or you would like to continue working on it, please feel free to re-open it and ask for a committer to remove the stale tag and review again.
Thanks all for your contribution.

@github-actions github-actions bot added the Stale label Dec 9, 2025
@github-actions github-actions bot closed this Dec 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mukund-thakur mukund-thakur mukund-thakur left review comments

Assignees

No one assigned

Labels

Stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@matsushita-shin @hadoop-yetus @steveloughran @mukund-thakur