GH-39262: [C++][Azure][FS] Add default credential auth configuration #39263

Tom-Newton · 2023-12-17T16:08:03Z

Rationale for this change

Default credential is a useful auth option.

What changes are included in this PR?

Implement AzureOptions::ConfigureDefaultCredential plus a little bit of plumbing to go around it.
Created a simple test.

Are these changes tested?

Added a simple unittest that everything initialises happily. This does not actually test a successful authentication. I think to do a real authentication with Azure we would need to run the test against real blob storage and we would need to create various identities which are non-trivial to create. Personally I think this is ok because all the complexity is abstracted away by the Azure SDK.

Are there any user-facing changes?

Closes: [C++][FS][Azure] Default credential authentication #39262

cpp/src/arrow/filesystem/azurefs_test.cc

cpp/src/arrow/filesystem/azurefs.h

kou · 2023-12-17T21:29:55Z

cpp/src/arrow/filesystem/azurefs_test.cc

+  options.backend = AzureBackend::kAzurite;  // Irrelevant for this test because it
+                                             // doesn't connect to the server.
+  ARROW_EXPECT_OK(options.ConfigureDefaultCredential("dummy-account-name"));
+  EXPECT_OK_AND_ASSIGN(auto default_credential_fs, AzureFileSystem::Make(options));


It seems that we can use Azurite with the DefaultAzureCredential: https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=visual-studio%2Cblob-storage#azure-sdks

Can we do an operation (CreateDir()?) and check the result to verify whether this filesystem is valid or not?

I had a bit of a try but I couldn't get the SSL setup working. I'll give it another go but I don't have great hope for making it work in CI even if I get it working locally.

OK. I'll also try it.

I tried and understand why you said SSL.
Azure::Identity::DefaultAzureCredential uses a Bearer token and Azure SDK for C++ rejects it with http: Bearer token authentication is not permitted for non TLS protected (https) endpoints.
If we want to use DefaultAzureCredential with Azurite, we need to generate a key and certificate pair and use it.

I looked at how to set it to Azure SDK for C++. It seems that we need to BlobClientOptions::Transport::Transport:

https://github.com/Azure/azure-sdk-for-cpp/blob/e328665588d2998fea198972af90d5108e64a968/sdk/storage/azure-storage-blobs/inc/azure/storage/blobs/blob_options.hpp#L174

https://github.com/Azure/azure-sdk-for-cpp/blob/e328665588d2998fea198972af90d5108e64a968/sdk/core/azure-core/inc/azure/core/internal/client_options.hpp#L97

https://github.com/Azure/azure-sdk-for-cpp/blob/e328665588d2998fea198972af90d5108e64a968/sdk/core/azure-core/inc/azure/core/http/policies/policy.hpp#L221

If we set BlobClientOptions::Transport::Transport, we need to specify curl based HTTP transport implementation or WinHTTP based HTTP transport implementation. They have different configurations for TLS...

How about using TestAzureHierarchicalNSFileSystem to test DefaultAzureCredential? If we use the real Azure service, we don't need to custom TLS configuration.

It would definitely be possible to test against real blob storage but there will be a significant amount of manual configuration for all the identities to test all the different authentications.

Then we need to provide details of these identities to TestAzureHierarchicalNSFileSystem either they are required always or we need to add new versions for each auth e.g. TestAzureHierarchicalNSFileSystemWithServicePrincipal.

Also there are some Auth methods that are not feasible to test. For example managed identity can only work on Azure VMs and workload identity can only work in kubernetes.

Personally I don't think this is worthwhile to make a more comprehensive test because of how little complexity there is outside the Azure SDK.

I think this test is enough, because as @pitrou said the other day: "we are not re-implementing the Azure SDK".

Also there are some Auth methods that are not feasible to test. For example managed identity can only work on Azure VMs and workload identity can only work in kubernetes.

Oh...

OK. I see.

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

kou

+1

kou · 2023-12-19T07:31:28Z

cpp/src/arrow/filesystem/azurefs_test.cc

+  options.backend = AzureBackend::kAzurite;  // Irrelevant for this test because it
+                                             // doesn't connect to the server.
+  ARROW_EXPECT_OK(options.ConfigureDefaultCredential("dummy-account-name"));
+  EXPECT_OK_AND_ASSIGN(auto default_credential_fs, AzureFileSystem::Make(options));


Also there are some Auth methods that are not feasible to test. For example managed identity can only work on Azure VMs and workload identity can only work in kubernetes.

Oh...

OK. I see.

conbench-apache-arrow · 2023-12-19T15:35:00Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 659b231.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

…39319) ### Rationale for this change Workload identity is a useful Azure authentication method. ### What changes are included in this PR? Implement `AzureOptions::ConfigureWorkloadIdentityCredential` ### Are these changes tested? Added a simple test initialising a fileystem using `ConfigureWorkloadIdentityCredential`. This is not the most comprehensive test but its the same as what we agreed on for #39263. ### Are there any user-facing changes? Workload identity authentication is now supported. * Closes: #39318 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…39321) ### Rationale for this change Workload identity is a useful Azure authentication method. Also I failed to set the account_name correctly for a bunch of auths (I think this got lost in a rebase then I copy pasted the broken code). ### What changes are included in this PR? - Make filesystem initialisation fail if `account_name_.empty()`. This prevents the account name configuration bug we had. Also added a test asserting that filesystem initialization fails in this case. - Remove account name configuration on all auth configs, in favour of setting in separately from the auth configuration. - Implement `AzureOptions::ConfigureManagedIdentityCredential` ### Are these changes tested? Added a simple test initialising a filesystem using `ConfigureManagedIdentityCredential`. This is not the most comprehensive test but its the same as what we agreed on for #39263. ### Are there any user-facing changes? Managed identity authentication is now supported. * Closes: #39320 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>

…ation (apache#39263) ### Rationale for this change Default credential is a useful auth option. ### What changes are included in this PR? Implement `AzureOptions::ConfigureDefaultCredential` plus a little bit of plumbing to go around it. Created a simple test. ### Are these changes tested? Added a simple unittest that everything initialises happily. This does not actually test a successful authentication. I think to do a real authentication with Azure we would need to run the test against real blob storage and we would need to create various identities which are non-trivial to create. Personally I think this is ok because all the complexity is abstracted away by the Azure SDK. ### Are there any user-facing changes? * Closes: apache#39262 Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…tion (apache#39319) ### Rationale for this change Workload identity is a useful Azure authentication method. ### What changes are included in this PR? Implement `AzureOptions::ConfigureWorkloadIdentityCredential` ### Are these changes tested? Added a simple test initialising a fileystem using `ConfigureWorkloadIdentityCredential`. This is not the most comprehensive test but its the same as what we agreed on for apache#39263. ### Are there any user-facing changes? Workload identity authentication is now supported. * Closes: apache#39318 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…ion (apache#39321) ### Rationale for this change Workload identity is a useful Azure authentication method. Also I failed to set the account_name correctly for a bunch of auths (I think this got lost in a rebase then I copy pasted the broken code). ### What changes are included in this PR? - Make filesystem initialisation fail if `account_name_.empty()`. This prevents the account name configuration bug we had. Also added a test asserting that filesystem initialization fails in this case. - Remove account name configuration on all auth configs, in favour of setting in separately from the auth configuration. - Implement `AzureOptions::ConfigureManagedIdentityCredential` ### Are these changes tested? Added a simple test initialising a filesystem using `ConfigureManagedIdentityCredential`. This is not the most comprehensive test but its the same as what we agreed on for apache#39263. ### Are there any user-facing changes? Managed identity authentication is now supported. * Closes: apache#39320 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>

…ation (apache#39263) ### Rationale for this change Default credential is a useful auth option. ### What changes are included in this PR? Implement `AzureOptions::ConfigureDefaultCredential` plus a little bit of plumbing to go around it. Created a simple test. ### Are these changes tested? Added a simple unittest that everything initialises happily. This does not actually test a successful authentication. I think to do a real authentication with Azure we would need to run the test against real blob storage and we would need to create various identities which are non-trivial to create. Personally I think this is ok because all the complexity is abstracted away by the Azure SDK. ### Are there any user-facing changes? * Closes: apache#39262 Lead-authored-by: Thomas Newton <thomas.w.newton@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…tion (apache#39319) ### Rationale for this change Workload identity is a useful Azure authentication method. ### What changes are included in this PR? Implement `AzureOptions::ConfigureWorkloadIdentityCredential` ### Are these changes tested? Added a simple test initialising a fileystem using `ConfigureWorkloadIdentityCredential`. This is not the most comprehensive test but its the same as what we agreed on for apache#39263. ### Are there any user-facing changes? Workload identity authentication is now supported. * Closes: apache#39318 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…ion (apache#39321) ### Rationale for this change Workload identity is a useful Azure authentication method. Also I failed to set the account_name correctly for a bunch of auths (I think this got lost in a rebase then I copy pasted the broken code). ### What changes are included in this PR? - Make filesystem initialisation fail if `account_name_.empty()`. This prevents the account name configuration bug we had. Also added a test asserting that filesystem initialization fails in this case. - Remove account name configuration on all auth configs, in favour of setting in separately from the auth configuration. - Implement `AzureOptions::ConfigureManagedIdentityCredential` ### Are these changes tested? Added a simple test initialising a filesystem using `ConfigureManagedIdentityCredential`. This is not the most comprehensive test but its the same as what we agreed on for apache#39263. ### Are there any user-facing changes? Managed identity authentication is now supported. * Closes: apache#39320 Authored-by: Thomas Newton <thomas.w.newton@gmail.com> Signed-off-by: Felipe Oliveira Carvalho <felipekde@gmail.com>

github-actions bot added awaiting review Awaiting review Component: C++ labels Dec 17, 2023

Tom-Newton added 2 commits December 17, 2023 16:09

Implement default credential

927ff6d

Simple test

56d796f

Tom-Newton force-pushed the tomnewton/azure_default_credential/GH-39262 branch from eab9db9 to 56d796f Compare December 17, 2023 16:09

Don't configure azurite for test

4dbe947

Tom-Newton marked this pull request as ready for review December 17, 2023 16:25

kou reviewed Dec 17, 2023

View reviewed changes

github-actions bot added awaiting review Awaiting review awaiting changes Awaiting changes awaiting committer review Awaiting committer review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Dec 17, 2023

kou changed the title ~~GH-39262: [C++][Azure][FS] default credential auth~~ GH-39262: [C++][Azure][FS] Add default credential auth configuration Dec 18, 2023

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 18, 2023

Make SetUrlsForAccountName private

5e36f90

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 18, 2023

Tom-Newton and others added 2 commits December 18, 2023 13:21

American spelling

0d9ccab

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Auto-format

c2a8625

felipecrv approved these changes Dec 19, 2023

View reviewed changes

kou approved these changes Dec 19, 2023

View reviewed changes

kou merged commit 659b231 into apache:main Dec 19, 2023
36 of 37 checks passed

kou removed the awaiting change review Awaiting change review label Dec 19, 2023

github-actions bot added the awaiting merge Awaiting merge label Dec 19, 2023

This was referenced Dec 20, 2023

GH-39318: [C++][FS][Azure] Add workload identity auth configuration #39319

Merged

GH-39320: [C++][FS][Azure] Add managed identity auth configuration #39321

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-39262: [C++][Azure][FS] Add default credential auth configuration #39263

GH-39262: [C++][Azure][FS] Add default credential auth configuration #39263

Tom-Newton commented Dec 17, 2023 •

edited

kou Dec 17, 2023

Tom-Newton Dec 17, 2023

kou Dec 18, 2023

kou Dec 18, 2023

Tom-Newton Dec 18, 2023 •

edited

felipecrv Dec 19, 2023

kou Dec 19, 2023

kou left a comment

kou Dec 19, 2023

conbench-apache-arrow bot commented Dec 19, 2023

GH-39262: [C++][Azure][FS] Add default credential auth configuration #39263

GH-39262: [C++][Azure][FS] Add default credential auth configuration #39263

Conversation

Tom-Newton commented Dec 17, 2023 • edited

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

kou Dec 17, 2023

Choose a reason for hiding this comment

Tom-Newton Dec 17, 2023

Choose a reason for hiding this comment

kou Dec 18, 2023

Choose a reason for hiding this comment

kou Dec 18, 2023

Choose a reason for hiding this comment

Tom-Newton Dec 18, 2023 • edited

Choose a reason for hiding this comment

felipecrv Dec 19, 2023

Choose a reason for hiding this comment

kou Dec 19, 2023

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

kou Dec 19, 2023

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Dec 19, 2023

Tom-Newton commented Dec 17, 2023 •

edited

Tom-Newton Dec 18, 2023 •

edited