Request throttling while refreshing state

[brandontosch]

We've been using terraform as we migrate our infrastructure from private data centers in to azure. We're nearing completion and now that our terraform codebase has gotten quite large we've started running in to an issue where we are getting throttled by Azure when attempting to refresh state before a plan or apply. This occurs even when parallelism is set to one..

After capturing some of these failed requests, it seems that the Azure API is ending back a retry-after header but it seems to be ignored and terraform just errors out without waiting or trying again.

I know terraform is using the azure go sdk, but not certain if this is something that would need to be changed in the sdk itself, in the terraform azurerm provider or terraform core..

Terraform Version

0.9.11

Affected Resource(s)

Any

Expected Behavior

The retry-after header should be respected and terraform should wait the specified amount of time before proceeding to send more requests rather than just erroring out.

Actual Behavior

Terraform produces errors such as:

* module.rpsql0.azurerm_storage_account.backups: 1 error(s) occurred:

* module.rpsql0.azurerm_storage_account.backups: azurerm_storage_account.backups: Error reading the state of AzureRM Storage Account "usw80cavrpsql0ragrs": storage.AccountsClient#GetProperties: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled."

Here is the response received from Azure:

HTTP/1.1 429
Cache-Control: no-cache
Pragma: no-cache
Content-Length: 80
Content-Type: application/json
Expires: -1
Retry-After: 17
x-ms-request-id: ***
Server: Microsoft-Azure-Storage-Resource-Provider/1.0
Server: Microsoft-HTTPAPI/2.0
x-ms-ratelimit-remaining-subscription-reads: 14734
x-ms-correlation-request-id: ***
x-ms-routing-request-id: ***
Strict-Transport-Security: max-age=31536000; includeSubDomains
Date: Fri, 28 Jul 2017 21:20:41 GMT
Connection: close

{"error":{"code":"TooManyRequests","message":"The request is being throttled."}}

Steps to Reproduce

Well that's a little tricky. Need a pretty decent sized codebase and unfortunately we can't really share ours due to company policy. Might be able to reproduce with a smaller sized codebase by using higher parallelism, but not sure.

[tombuildsstuff]

Hey @brandontosch

Thanks for opening this issue :)

Support for automatically retrying Retryable errors was added to the Azure SDK for Go in Azure/go-autorest#155 which shipped in v0.1.6 of the AzureRM Terraform Provider.

Would you be able to upgrade to the latest version of the AzureRM Terraform Provider and let us know if you're still seeing this issue?

Thanks!

[tombuildsstuff]

ping @brandontosch :)

[brandontosch]

Hi @tombuildsstuff!

Sorry for the delay, was out last couple of days. It does seem to be much better now; we can see the requests start to slow down after a while, but unfortunately we still hit some throttling issues intermittently.

Here's some snippets from our last run:

Initializing the backend...

Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "azurerm" (0.1.7)...
- Downloading plugin for provider "ultradns" (0.1.0)...
- Downloading plugin for provider "vault" (0.1.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.azurerm: version = "~> 0.1"
* provider.ultradns: version = "~> 0.1"
* provider.vault: version = "~> 0.1"

Terraform has been successfully initialized!

...

Error refreshing state: 55 error(s) occurred:

* module.suweb.azurerm_storage_container.role: 1 error(s) occurred:

* module.suweb.azurerm_storage_container.role: azurerm_storage_container.role: Error retrieving keys for storage account "***": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled."
* module.ret.azurerm_storage_container.role: 1 error(s) occurred:

* module.ret.azurerm_storage_container.role: azurerm_storage_container.role: Error retrieving keys for storage account "***": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled."
* module.swweb.azurerm_storage_blob.1_configure_puppet: 1 error(s) occurred:

* module.swweb.azurerm_storage_blob.1_configure_puppet: azurerm_storage_blob.1_configure_puppet: Error retrieving keys for storage account "***": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled."
* module.ntp.azurerm_storage_blob.0_bootstrap: 1 error(s) occurred:

* module.ntp.azurerm_storage_blob.0_bootstrap: azurerm_storage_blob.0_bootstrap: Error retrieving keys for storage account "***": storage.AccountsClient#ListKeys: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled."

[JunyiYi]

Hi @brandontosch , can you please verify whether this issue is resolved? Thanks.

[brandontosch]

@JunyiYi I can confirm that we have not seen this issue in some time now so I would be comfortable saying it is resolved at this point.

[metacpp]

Thanks @brandontosch for the confirmation, we will close this issue now.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!