Error while list service accounts if access to some of them is denied

[AntonDobshikov]

Description

We have Azure Databricks in our subscription and the plugin is unable to list storage accounts
MS Support says that it's designed that Azure Databricks creates dey assignment for its service account and nothing can be done from Azure side here.

Environment

TeamCity Enterprise 2020.1.5 (build 78938)
Azure Resource Manager Cloud Support SNAPSHOT-20200312092542

Diagnostic logs

teamcity-clouds.log

[AntonDobshikov]

#126

[mikeclayton]

Same issue here with these versions:

• TeamCity Enterprise 2020.2.2 (build 85899)
• Azure Resource Manager Cloud Support 9.6.0

With the "debug-cloud" logging preset enabled we see this in our teamcity.clouds.log:

[2021-06-14 11:16:03,761]  DEBUG [atcher-worker-3] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] Trying to get data from cache for task FetchStorageAccounts, mode Normal.
[2021-06-14 11:16:03,761]  DEBUG [atcher-worker-3] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] There is no data in cache for task FetchStorageAccounts, mode Normal. Adding request to queue. Time to start: 2021-06-14T10:16:03.761349
[2021-06-14 11:16:03,761]  DEBUG [o-8111-exec-157] - onnector.AzureApiConnectorImpl - Received list of vm sizes in region northeurope
[2021-06-14 11:16:03,783]  DEBUG [o-8111-exec-157] - onnector.AzureApiConnectorImpl - Received list of networks
[2021-06-14 11:16:03,986]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] Start executing batch of tasks. TaskId: FetchStorageAccounts, Task count: 1
[2021-06-14 11:16:05,057]  DEBUG [ueue executor 1] - tler.AzureThrottlerInterceptor - [7-ReadAdapter] Azure request processed: Remaining reads: 11981, Url: https://management.azure.com/subscriptions/[my-subscription-id]/providers/Microsoft.Storage/storageAccounts?api-version=2019-06-01
[2021-06-14 11:16:05,057]  DEBUG [ueue executor 1] - tler.AzureThrottlerInterceptor - [7-ReadAdapter] Azure request processed: Requests sequence length: 0
[2021-06-14 11:16:05,057]  DEBUG [ueue executor 1] - tler.AzureThrottlerInterceptor - [7-ReadAdapter] Azure request processed: Headers: x-ms-ratelimit-remaining-subscription-reads=11981, x-ms-ratelimit-remaining-subscription-resource-requests=null, x-ms-ratelimit-remaining-tenant-reads=null, x-ms-ratelimit-remaining-tenant-resource-requests=null, 
[2021-06-14 11:16:05,414]  DEBUG [ueue executor 1] - tler.AzureThrottlerInterceptor - [7-ReadAdapter] Azure request processed: Remaining reads: null, Url: https://management.azure.com/subscriptions/[my-subscription-id]/resourceGroups/[my-databricks-rg]/providers/Microsoft.Storage/storageAccounts/dbstorageabcdefghijklm/listKeys?api-version=2019-06-01
[2021-06-14 11:16:05,414]  DEBUG [ueue executor 1] - tler.AzureThrottlerInterceptor - [7-ReadAdapter] Azure request processed: Requests sequence length: 1
[2021-06-14 11:16:05,414]  DEBUG [ueue executor 1] - tler.AzureThrottlerInterceptor - [7-ReadAdapter] Azure request processed: Headers: x-ms-ratelimit-remaining-subscription-reads=null, x-ms-ratelimit-remaining-subscription-resource-requests=null, x-ms-ratelimit-remaining-tenant-reads=null, x-ms-ratelimit-remaining-tenant-resource-requests=null, 
[2021-06-14 11:16:05,416]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] [FetchStorageAccounts] Received task notification. Kind: OnError
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - ler.AzureThrottlerStrategyImpl - Trying to reset cache timeout for periodical task FetchResourceGroups
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] New timeout 0 was ignored for FetchResourceGroups task (current value: 150)
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - ler.AzureThrottlerStrategyImpl - Trying to reset cache timeout for periodical task FetchServices
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] New timeout 0 was ignored for FetchServices task (current value: 150)
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - ler.AzureThrottlerStrategyImpl - Trying to set cache timeout for periodical task FetchCustomImages to 1 sec
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] New timeout 1 was ignored for FetchCustomImages task (current value: 90)
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - ler.AzureThrottlerStrategyImpl - Trying to set cache timeout for periodical task FetchInstances to 10 sec
[2021-06-14 11:16:05,418]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] New timeout 10 was ignored for FetchInstances task (current value: 150)
[2021-06-14 11:16:05,419]  DEBUG [ueue executor 1] - er.AzureThrottlerTaskQueueImpl - [7-ReadAdapter] Start executing batch of tasks. TaskId: FetchCustomImages, Task count: 1
[2021-06-14 11:16:05,419]  DEBUG [o-8111-exec-157] - onnector.AzureApiConnectorImpl - Failed to get list of storage accounts in region northeurope: OnError while emitting onNext value: com.microsoft.azure.management.storage.implementation.StorageAccountImpl.class
rx.exceptions.OnErrorThrowable$OnNextValue: OnError while emitting onNext value: com.microsoft.azure.management.storage.implementation.StorageAccountImpl.class

...

[2021-06-14 11:16:05,421]   INFO [o-8111-exec-157] - ure.arm.web.SettingsController - Failed to process storageAccounts request: jetbrains.buildServer.clouds.CloudException: Failed to get list of storage accounts in region northeurope: OnError while emitting onNext value: com.microsoft.azure.management.storage.implementation.StorageAccountImpl.class
[2021-06-14 11:16:05,421]  DEBUG [o-8111-exec-157] - ure.arm.web.SettingsController - Failed to process storageAccounts request
jetbrains.buildServer.clouds.CloudException: Failed to get list of storage accounts in region northeurope: OnError while emitting onNext value: com.microsoft.azure.management.storage.implementation.StorageAccountImpl.class

The issue seems to be that the plugin enumerates the storage accounts in the subscription fine, but then then it tries to invoke the "listKeys" action on each, but it doesn't have permission to storage accounts in databricks resource groups.

This is probably an issue more generally with "deny" permissions on any storage accounts, but it's surfacing here with databricks specifically.

Note the error in 0.9.6 seems to have less information than the OP's version "SNAPSHOT-20200312092542".

[mikeclayton]

See also https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360010664580-Cloud-profile-creation-issues for another affected user.