backend/azure: terraform output returns blank results although apply returns correct output

[panshow81]

Terraform Version

Terraform v0.12.29 
+ provider.azurerm v2.15.0

Terraform Configuration Files

terraform {  
  backend "azurerm" {
       resource_group_name      = "###" 
       storage_account_name	= "#####" 
       container_name	        = "#####" 
       key                      = "terraform-test/test-dev.tfstate" 
       sas_token                = ####
  }
}

module "vm" {
    source               = "git@github.bnsf.com:DC/terraform-azure-windows-vm.git"
    environment          = "${var.environment}"
    location             = "${var.location}"
    resource_group_name  = "${var.resource_group_name}"
    virtual_network_name = "${var.virtual_network_name}"
    vnet_rg_name         = "${var.vnet_rg_name}"
    subnet_name          = "${var.subnet_name}"
    server_prefix        = "${var.server_prefix}" 
    tags                 = "${var.tags}"
    vm_size              = "${var.vm_size}"
    server_count         = "${var.server_count}"  
}
  
module "vm_extension" {
    source               = "git@github.bnsf.com:DC/terraform-azure-vm-extension.git"
    virtual_machineid    = module.vm.virtual_machine_id
    server_count         = "${var.server_count}"
    file_uri             = "${var.file_uri}"
    command_to_execute   = "${var.command_to_execute}"
}

Debug Output

Crash Output

Expected Behavior

terraform output -json should return

{
  vm_host_names: {
    sensitive: false,
    type: [
      "tuple",
      [
        "string"
      ],
      
    ],
    value: [
      "patup-dev-vm001"
    ],    
  },
  vm_ips: {
    sensitive: false,
    type: [
      "tuple",
      [
        "string"
      ],
      
    ],
    value: [
      "10.222.7.4"
    ],
    
  },  
}

Actual Behavior

Steps to Reproduce

1. terraform init --input=false -backend=true -reconfigure
2. terraform plan --input=false -out tf-dev.plan -var-file=./dev_variables.tfvars
3. terraform apply tf-dev.plan print below log

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

vm_host_names = [
"patup-dev-vm001",
]
vm_ips = [
"10.222.7.5",
]

4. terraform output -json return blank JSON

Additional Context

Whenever this issue occurs the state file in blob storage was not updated with the correct state, apply commands output says 5 resources are created although the state file doesn't get updated, state file remains as no resources are created and that's the reason I think terraform output returns blank.
This issue is not 100% reproducible, it only occurs 15-20% times, I think sometimes state file is not getting updated even though apply command executes successfully and returns the appropriate result.

Also tried running this with local backend instead of remote (azurerm), in the case of local backend terraform output never fails to returns expected JSON.

Question:
Why terraform apply doesn't fail if the state file is not updated? or how to debug the issue and find the root cause as it not reproducible 100% of the time its difficult to debug it.

References

[alisdair]

Given that the local backend never has this problem, this seems like it might be an issue with Azure's storage service being eventually consistent. This used to be an issue with the S3 backend (fixed in #14746), and from a brief glance doesn't appear to be addressed by the Azure backend.

If you wait a significant time (several minutes) between running terraform apply and terraform output, does the problem still occur?

[panshow81]

I tried with adding a min or 2 mins delay not with adding several minutes of delay between terraform apply and terraform output. i didn't get any different results when 1-2 mins delay was added.

[tombuildsstuff]

@panshow81 there's been some significant changes to the Azure Backend starting in Terraform 0.13 - are you able to confirm if this is reproducible using Terraform 0.13 or above?

Thanks!

[panshow81]

@tombuildsstuff : I tried to run it with terraform 0.13.5 but terraform is crashing on init.
with terraform 0.13.5, it works perfectly fine without azurerm backed, although it crashes with azurerm backend.

panic: runtime error: makeslice: len out of range goroutine 1 [running]: github.com/tombuildsstuff/giovanni/storage/2018-11-09/blob/blobs.Client.GetResponder(0x2cda2a0, 0xc000530360, 0x2cdf8e0, 0xc000526420, 0x0, 0x0, 0xdf8475800, 0x34630b8a000, 0x3, 0x6fc23ac00, ...) /home/circleci/project/project/vendor/github.com/tombuildsstuff/giovanni/storage/2018-11-09/blob/blobs/get.go:104 +0x223 github.com/tombuildsstuff/giovanni/storage/2018-11-09/blob/blobs.Client.Get(0x2cda2a0, 0xc000530360, 0x2cdf8e0, 0xc000526420, 0x0, 0x0, 0xdf8475800, 0x34630b8a000, 0x3, 0x6 fc23ac00, ...) /home/circleci/project/project/vendor/github.com/tombuildsstuff/giovanni/storage/2018-11-09/blob/blobs/get.go:61 +0x378 github.com/hashicorp/terraform/backend/remote-state/azure.(*RemoteClient).Get(0xc0004560f0, 0xc000207890, 0x418c1e0, 0x272c535) /home/circleci/project/project/backend/remote-state/azure/client.go:40 +0x131 github.com/hashicorp/terraform/states/remote.(*State).refreshState(0xc000536b40, 0x7, 0x0) /home/circleci/project/project/states/remote/state.go:111 +0x3f github.com/hashicorp/terraform/states/remote.(*State).RefreshState(0xc000536b40, 0x0, 0x0) /home/circleci/project/project/states/remote/state.go:104 +0x73 github.com/hashicorp/terraform/command.(*InitCommand).Run(0xc0002361c0, 0xc00013a020, 0x4, 0x6, 0xc0005062a0) /home/circleci/project/project/command/init.go:278 +0x16af github.com/mitchellh/cli.(*CLI).Run(0xc00029c500, 0xc00029c500, 0xc000347d48, 0x1) /home/circleci/project/project/vendor/github.com/mitchellh/cli/cli.go:255 +0x1e1 main.wrappedMain(0x0) /home/circleci/project/project/main.go:275 +0xc97 main.realMain(0x0) /home/circleci/project/project/main.go:106 +0xe5 main.main() /home/circleci/project/project/main.go:42 +0x41 !!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!! Terraform crashed! This is always indicative of a bug within Terraform. A crash log has been placed at "crash.log" relative to your current working directory. It would be immensely helpful if you could please report the crash with Terraform[1] so that we can fix this. When reporting bugs, please include your terraform version. That information is available on the first line of crash.log. You can also get it by running 'terraform --version' on the command line. SECURITY WARNING: the "crash.log" file that was created may contain sensitive information that must be redacted before it is safe to share on the issue tracker. [1]: https://github.com/hashicorp/terraform/issues !!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

[mildwonkey]

This issue was fixed with the merged PR so I am going to close it - thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.