Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement AAD authentication for Azure Storage #38803

Closed
wants to merge 2 commits into from
Closed

Implement AAD authentication for Azure Storage #38803

wants to merge 2 commits into from

Conversation

c-w
Copy link

@c-w c-w commented Mar 23, 2020

Summary

Currently, the Azure implementation of ActiveStorage supports shared key authentication. This commit extends the functionality to also support authentication via Azure Active Directory (AAD).

Other Information

Authenticating to Azure Storage via AAD enables using role-based access control (RBAC) to manage access to the storage resources. This enables multi-tenancy use-cases, such as for example having
multiple Rails applications use different containers in the same Azure Storage account while ensuring that each application can only access data in its own container.

To enable AAD RBAC authentication, configure the tenant ID, client ID and client secret of the AAD principal that should be used for authentication:

azure:
  service: AzureStorage
  storage_account_name: mystorageaccount
  container: mycontainer
  tenant_id: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  client_id: bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
  client_secret: <%= Rails.application.credentials.dig(:azure, :client_secret) %>

Note that the AAD principal used for authentication must be granted at least the roles "Storage Blob Delegator" on the storage account as well as "Storage Blob Data Contributor" on the container. The former role is required for generating shared access signatures for direct uploads and asset URLs and the latter role is required for read/write/delete/list permissions on the contents of the container.

Also note that the AAD authentication flow relies on user delegation keys which were introduced in Azure Storage API version 2018-11-09. This means that as of update 1811 the functionality will not yet work when targeting Azure Stack Hub Storage.

For testing purposes, all the required resources and permissions to exercise the AAD authentication flow can be set up with the following script (assuming the Azure CLI and JQ are installed):

service_principal_name="rails-azure-aad-integration-tests"  # CHANGE ME
location="eastus"  # CHANGE ME
subscription="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"  # CHANGE ME
storage_account_name="railsazureaad"  # CHANGE ME
container_name="integrationtests"  # CHANGE ME

az login
az account set -s "${subscription}"
az group create -l "${location}" -n "${resource_group_name}"
az storage account create -l "${location}" -g "${resource_group_name}" -n "${storage_account_name}" --kind StorageV2
az storage container create -n "${container_name}" --connection-string "$(az storage account show-connection-string -n "${storage_account_name}")"

sp="$(az ad sp create-for-rbac --name "${service_principal_name}" --skip-assignment)"
sp_tenant="$(jq -r '.tenant' <<< "${sp}")"
sp_client="$(jq -r '.appId' <<< "${sp}")"
sp_secret="$(jq -r '.password' <<< "${sp}")"

while ! az role assignment create --role "Storage Blob Data Contributor" --assignee "${sp_client}" --scope "/subscriptions/${subscription}/resourceGroups/${resource_group_name}/providers/Microsoft.Storage/storageAccounts/${storage_account_name}/blobServices/default/containers/${container_name}"; do sleep 2s; done

while ! az role assignment create --role "Storage Blob Delegator" --assignee "${sp_client}" --scope "/subscriptions/${subscription}/resourceGroups/${resource_group_name}/providers/Microsoft.Storage/storageAccounts/${storage_account_name}"; do sleep 2s; done

cat >> activestorage/test/service/configurations.yml << EOF
azure_aad:
  service: AzureStorage
  storage_account_name: "${storage_account_name}"
  tenant_id: "${sp_tenant}"
  client_id: "${sp_client}"
  client_secret: "${sp_secret}"
  container: "${container_name}"
EOF

Validated by @michaelperel
CC @jrodbeta

@blob_service = Azure::Storage::Blob::BlobService.create(storage_account_name: storage_account_name, storage_access_key: storage_access_key, **options)
@shared_access_signature = Azure::Storage::Common::Core::Auth::SharedAccessSignature.new(storage_account_name, storage_access_key)
end
end
Copy link
Contributor

@georgeclaghorn georgeclaghorn Apr 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I’ll have more feedback on these polymorphic clients, but for starters, let’s break them into separate files:

  • lib/active_storage/service/azure_storage_service/active_directory_client.rb (and remove Azure from the class name; the namespace makes it clear that it applies to Azure)
  • lib/active_storage/service/azure_storage_service/access_key_client.rb

Copy link
Author

@c-w c-w May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a689cbb

# container: ""
# tenant_id: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
# client_id: bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
# client_secret: cccccccc-cccc-cccc-cccc-cccccccccccc
Copy link
Contributor

@georgeclaghorn georgeclaghorn Apr 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any pointers on setting this up for CI? I’m pretty unfamiliar with Azure in general.

Copy link
Author

@c-w c-w May 1, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you already have a storage account you're using on CI? If so, all we need to do is create a service principal and grant it the appropriate IAM permissions to the storage account.

You can do this via a snippet similar to the one I posted in the PR description :

service_principal_name="enter some name here"
storage_account_name="enter your storage account name here"
container_name="enter the name for the container used for the new tests here"

# fetch the resource id of the existing storage account
storage_account_id="$(az storage account show --name "${storage_account_name}" --query id --output tsv)"

# create the container that the new tests will use
# we need to do this as a one-time setup step since for extra security we'll only grant
# our service principal access to this specific container instead of the entire storage account
# which means that the service principal won't be able to dynamically create new containers
az storage container create --name "${container_name}" --account-name "${storage_account_name}"

# create the service principal we'll be using for the tests
sp="$(az ad sp create-for-rbac --name "${service_principal_name}" --skip-assignment --output json)"
sp_tenant="$(jq -r '.tenant' <<< "${sp}")"
sp_client="$(jq -r '.appId' <<< "${sp}")"
sp_secret="$(jq -r '.password' <<< "${sp}")"

# grant the service principal the appropriate IAM permissions on the storage account
# the data contributor role is required for read/write/list/delete operations on blobs
# in the container and the delegator permission is required to create signed private urls
az role assignment create --role "Storage Blob Data Contributor" --assignee "${sp_client}" --scope "${storage_account_id}/blobServices/default/containers/${container_name}"
az role assignment create --role "Storage Blob Delegator" --assignee "${sp_client}" --scope "${storage_account_id}"

# last but not least, use these values to update configurations.yml
cat << EOM
azure_aad:
  service: AzureStorage
  storage_account_name: "${storage_account_name}"
  tenant_id: "${sp_tenant}"
  client_id: "${sp_client}"
  client_secret: "${sp_secret}"
  container: "${container_name}"
EOM

Alternatively you can also do this in the Azure Portal: see the docs for creating a service principal and the docs for assigning a storage role.

If you do not already have a storage account that's being used in the CI or wish to create a new one, you can use the snippet that I posted in the pull request description to create all the required resources and update the configurations.yml file as required.

Hope this helps. Let me know what additional assistance I can provide.

c-w added 2 commits Jul 19, 2020
Currently, the Azure implementation of ActiveStorage supports shared
key authentication [1]. This commit extends the functionality to also
support authentication via Azure Active Directory (AAD) [2].

Authenticating to Azure Storage via AAD enables using role-based
access control (RBAC) to manage access to the storage resources.
This enables multi-tenancy use-cases, such as for example having
multiple Rails applications use different containers in the same
Azure Storage account while ensuring that each application can
only access data in its own container.

To enable AAD RBAC authentication, configure the tenant ID, client ID
and client secret of the AAD principal that should be used for
authentication:

```yaml
azure:
  service: AzureStorage
  storage_account_name: mystorageaccount
  container: mycontainer
  tenant_id: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
  client_id: bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb
  client_secret: <%= Rails.application.credentials.dig(:azure, :client_secret) %>
```

Note that the AAD principal used for authentication must be granted at
least the roles "Storage Blob Delegator" on the storage account as well
as "Storage Blob Data Contributor" on the container. The former role is
required for generating shared access signatures for direct uploads and
asset URLs and the latter role is required for read/write/delete/list
permissions on the contents of the container.

Also note that the AAD authentication flow relies on user delegation
keys [3] which were introduced in Azure Storage API version 2018-11-09.
This means that as of update 1811 the functionality will not yet work
when targeting Azure Stack Hub Storage.

For testing purposes, all the required resources and permissions to
exercise the AAD authentication flow can be set up with the following
script:

```bash
service_principal_name="rails-azure-aad-integration-testes"  # CHANGE ME
location="eastus"  # CHANGE ME
subscription="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"  # CHANGE ME
storage_account_name="railsazureaad"  # CHANGE ME
container_name="integrationtests"  # CHANGE ME

az login
az account set -s "${subscription}"
az group create -l "${location}" -n "${resource_group_name}"
az storage account create -l "${location}" -g "${resource_group_name}" -n "${storage_account_name}" --kind StorageV2
az storage container create -n "${container_name}" --connection-string "$(az storage account show-connection-string -n "${storage_account_name}")"

sp="$(az ad sp create-for-rbac --name "${service_principal_name}" --skip-assignment)"
sp_tenant="$(jq -r '.tenant' <<< "${sp}")"
sp_client="$(jq -r '.appId' <<< "${sp}")"
sp_secret="$(jq -r '.password' <<< "${sp}")"

while ! az role assignment create --role "Storage Blob Data Contributor" --assignee "${sp_client}" --scope "/subscriptions/${subscription}/resourceGroups/${resource_group_name}/providers/Microsoft.Storage/storageAccounts/${storage_account_name}/blobServices/default/containers/${container_name}"; do sleep 2s; done

while ! az role assignment create --role "Storage Blob Delegator" --assignee "${sp_client}" --scope "/subscriptions/${subscription}/resourceGroups/${resource_group_name}/providers/Microsoft.Storage/storageAccounts/${storage_account_name}"; do sleep 2s; done

cat >> activestorage/test/service/configurations.yml << EOF
azure_aad:
  service: AzureStorage
  storage_account_name: "${storage_account_name}"
  tenant_id: "${sp_tenant}"
  client_id: "${sp_client}"
  client_secret: "${sp_secret}"
  container: "${container_name}"
EOF
```

[1] https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key
[2] https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-azure-active-directory
[3] https://docs.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key
@c-w c-w requested a review from georgeclaghorn Jul 19, 2020
@c-w
Copy link
Author

@c-w c-w commented Jul 19, 2020

@georgeclaghorn Could you take another look at this pull request? Are there any additional questions or concerns I can help address and/or pointers to provide for setting up the integration tests in CI?

I'm not 100% certain, but the latest test failures don't seem related to this pull request (and indeed at one point when I originally opened the pull request I recall that the build passed). Could you clarify this and/or provide some pointers for me to debug the failures? Thanks in advance!

@rails-bot
Copy link

@rails-bot rails-bot bot commented Oct 17, 2020

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Thank you for your contributions.

@rails-bot rails-bot bot added the stale label Oct 17, 2020
@rails-bot rails-bot bot closed this Oct 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants