Terraform Plan/Apply unable to read state file from another subscription

[slawchod]

Hi,

We have created central storage account in separated subscription to store state files in secure location. In single stage we have terraform init which is using service connection A to subscription A (for backend setup) and in next step we are having terraform plan or apply with service connection B to subscription B. Additionally SPN under B has rights to storage account from A.

Such setup was working for couple of months but after 25th of May (more or less) something has changed and we have started to get following errors on plan/apply.

Error: Error loading state: Error retrieving keys for Storage Account "<storage_account_name>": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group '<resource_group_name>' could not be found

Init itself was successful all the time. There was no code change on our side. We even extended permissions for tests and we set Contributor for B SPN to entire A subscription but issue was the same.

The strange thing is that successful (prior issue) and failed steps have the same Terraform extension version: 0.0.142. We install Terraform version 0.12.3.

We have some kind of workaround but it would require reconfiguration in many release pipelines and tens of commits. If you could revert change in extension it would be great. Thanks!

[cuzza0]

@slawchod Are you able to detail the workaround you have in place? I am having a similar issue, would be good to see what options there are.

[slawchod]

Hi @cuzza0,

No problem. So in Azure DevOps within terraform init you are specifying all terraform backend details. Previously it was sufficient for terraform plan/apply, so in code it was like that:

terraform {
     backend "azurerm" {}
 }

Now, when terraform backend settings from Azure DevOps are not taken fully from init to plan/apply I've changed code to include subscription details in backend config in the code as well:

terraform {
     backend "azurerm" {
       subscription_id = "#{tfbeSubscriptionId}#"
     }
 }

You might have a static value for subscirption_id, but I prefer to have it as variable and perform replace token task as the fist step in the stage.

I hope it will help :)

Cheers!

[vparmeland]

Hi

I recently faced the same issue "Terraform: Allow state files to be stored in a different subscription" microsoft/azure-pipelines-extensions#707

I'm trying to deploy Azure resources and states in different subscriptions too :

• 1 Central Subscription to store tfstates
• "n" subscriptions to deploy Azure resources

I had the same 404 issue before, thanks @slawchod :1

terraform {
     backend "azurerm" {
       subscription_id = "#{tfbeSubscriptionId}#"
     }
 }

Now I'm getting a 403 Error on my "tsfate" storage container
Error: Error loading state: Error retrieving keys for Storage Account "XXXXXX": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'xxxxxxxxxxxxxxxxxxxx' with object id 'xxxxxxxxxxxxxxxxxxxx does not have authorization to perform action 'Microsoft.Storage/storageAccounts/listKeys/action' over scope '/subscriptions/ssssssssssssssssssssssssss/resourceGroups/My-Central-States-123/providers/Microsoft.Storage/storageAccounts/centralstorageblabla' or the scope is invalid. If access was recently granted, please refresh your credentials."

Azure Pipeline YML (Terraform Extension) : Despite using the parameter "backendServiceArm" (The Service Connection used for my TFStates Subscription & RG) Terraform is still using the parameter "environmentServiceNameAzureRM" (The Azure Environnement to deploy resource. This SP should not have the privileges to the TFSates Sub/RG)

Find below my Azure Pipeline Task yml (Terraform extension)

    - task: TerraformTaskV1@0
      displayName: 'Terraform Plan'
      name: terraformPlan
      inputs:
        provider: 'azurerm'
        command: 'plan'
        environmentServiceNameAzureRM: '#### SP used to deploy Azure Resources in my dev subscription**'
        backendServiceArm: '#### SP used to centralize Tfstates in my backend subscription'
        backendAzureRmResourceGroupName: 'My-Central-States-123'
        backendAzureRmStorageAccountName: 'mycentralstorage'
        backendAzureRmContainerName: 'mycentral-container-states'
        backendAzureRmKey: 'terraform.tfstate'
        workingDirectory: .......................

I tried several configurations... Do you have an idea?

Thanks

[guidooliveira]

Trying to use a different subscription for the backend on TerraformTaskV3@3 with commandOptions gets ignored/duplicated as it inserts subscription_id at the end using the default subscription of the SPN. Is there any other way to specify the subscription_id for the backend?

commandOptions: "-backend-config=subscription_id=$(backendAzureRmSubscriptionId)"

this same approach works if I set use_azure_ad to true on the backend and assign the Storage Blob Contributor to the SPN. Why can't I do the same with the default graph?

[mericstam]

The extension does not currently support having the state file in another subscription. Looking into what it would take to fix it

[guidooliveira]

The extension does not currently support having the state file in another subscription. Looking into what it would take to fix it

I've checked the azurerm handler source code and it always injects the backend-config value subscription_id from the service connection's, I believe adding another non-required value for the subscriptionId would be enough to handle it, if said parameter is null, assume the one from the service connection, otherwise, just use the supplied one.
the handling of the subscription_id can be observed here:

azure-pipelines-terraform/Tasks/TerraformTask/TerraformTaskV3/src/azure-terraform-command-handler.ts

Line 20 in 1a1673c

this.backendConfig.set('subscription_id', tasks.getEndpointDataParameter(backendServiceName, "subscriptionid", true));

As a workaround I'm using custom to run init and supply the values myself. It does seem to introduce a weird issue with terraform providers running before terraform plan in the plan command task. but I managed to have it running nonetheless.

[mericstam]

So after a bit of tinkering I manage to have two different service connections for separate subscriptions.
Just as the original comment I have the 'init' command on one service connection where I have my state files. backendServiceArm: 'myStatesSPN'
And on my 'plan' and 'apply' commands I set environmentServiceNameAzureRM: 'myProdSPN'
This way I have states and environments separated. The two different service connections are backed by totally different Azure Subs with separate AAD.
I would think you need one service connection for each subscription you wish to deploy towards. I only have two so I can't test deploy to multiple subs.

example:

- task: TerraformTaskV3@3
      displayName: 'terraform init'
      inputs:
        provider: 'azurerm'
        command: 'init'
        backendServiceArm: 'myStatesSPN'
        backendAzureRmResourceGroupName: 'terraform'
        backendAzureRmStorageAccountName: 'statestore'
        backendAzureRmContainerName: 'state'
        backendAzureRmKey: 'state.tfstate'

   - task: TerraformTaskV3@3
      name: plan
      displayName: 'terraform plan'
      inputs:
        provider: 'azurerm'
        command: 'plan'
        commandOptions: '-out=tfplan'
        environmentServiceNameAzureRM: 'myProdSPN'

[slawchod]

I agree that MS fixed that in TerraformTaskV3@3

[danielcgonzalez]

for me this is not a valid solution, we should be able of using the same service principal, to store tfstate in different subscription, at least, if we could replace the backend data with container id, for example