Retry Calls on HTTP 5XX Response Status Codes

[bflad]

Terraform Version

Terraform v0.11.13
+ provider.github v1.3.0

Affected Resource(s)

In our use case, mainly:

• github_issue_label

Terraform Configuration Files

# Terraform module used across dozens of repositories

resource "github_issue_label" "breaking-change" {
  repository = "${github_repository.provider.name}"
  name       = "breaking-change"
  color      = "d93f0b"
}

resource "github_issue_label" "bug" {
  repository = "${github_repository.provider.name}"
  name       = "bug"
  color      = "f7c6c7"
}

resource "github_issue_label" "crash" {
  repository = "${github_repository.provider.name}"
  name       = "crash"
  color      = "e11d21"
}

resource "github_issue_label" "documentation" {
  repository = "${github_repository.provider.name}"
  name       = "documentation"
  color      = "fef2c0"
}

resource "github_issue_label" "dependencies" {
  repository = "${github_repository.provider.name}"
  name       = "dependencies"
  color      = "fad8c7"
}

resource "github_issue_label" "enhancement" {
  repository = "${github_repository.provider.name}"
  name       = "enhancement"
  color      = "d4c5f9"
}

resource "github_issue_label" "good_first_issue" {
  repository = "${github_repository.provider.name}"
  name       = "good first issue"
  color      = "128A0C"
}

resource "github_issue_label" "hashibot_ignore" {
  repository = "${github_repository.provider.name}"
  name       = "hashibot/ignore"
  color      = "000000"
}

resource "github_issue_label" "help_wanted" {
  repository = "${github_repository.provider.name}"
  name       = "help wanted"
  color      = "128A0C"
}

resource "github_issue_label" "new-data-source" {
  repository = "${github_repository.provider.name}"
  name       = "new-data-source"
  color      = "d4c5f9"
}

resource "github_issue_label" "new-resource" {
  repository = "${github_repository.provider.name}"
  name       = "new-resource"
  color      = "d4c5f9"
}

resource "github_issue_label" "provider" {
  repository = "${github_repository.provider.name}"
  name       = "provider"
  color      = "bfd4f2"
}

resource "github_issue_label" "question" {
  repository = "${github_repository.provider.name}"
  name       = "question"
  color      = "cc317c"
}

resource "github_issue_label" "regression" {
  repository = "${github_repository.provider.name}"
  name       = "regression"
  color      = "e11d21"
}

resource "github_issue_label" "stale" {
  repository = "${github_repository.provider.name}"
  name       = "stale"
  color      = "e11d21"
}

resource "github_issue_label" "technical-debt" {
  repository = "${github_repository.provider.name}"
  name       = "technical-debt"
  color      = "1d76db"
}

resource "github_issue_label" "waiting-response" {
  repository = "${github_repository.provider.name}"
  name       = "waiting-response"
  color      = "5319e7"
}

resource "github_issue_label" "upstream" {
  repository = "${github_repository.provider.name}"
  name       = "upstream"
  color      = "fad8c7"
}

resource "github_issue_label" "upstream-terraform" {
  repository = "${github_repository.provider.name}"
  name       = "upstream-terraform"
  color      = "cccccc"
}

variable "sizes" {
  default = ["XS", "S", "M", "L", "XL", "XXL"]
}

resource "github_issue_label" "size" {
  count      = "${length(var.sizes)}"
  repository = "${github_repository.provider.name}"
  name       = "size/${var.sizes[count.index]}"
  color      = "ffffff"
}

Debug Output

Issue is intermittent, can provide if necessary.

Expected Behavior

The Terraform resource should retry the HTTP request on retryable HTTP response status codes (e.g. 5XX).

Actual Behavior

Error: Error refreshing state: 4 error(s) occurred:

* module.fortios.github_issue_label.stale: 1 error(s) occurred:

* module.fortios.github_issue_label.stale: github_issue_label.stale: GET https://api.github.com/repos/terraform-providers/terraform-provider-fortios/labels/stale: 502 Server Error []
* module.brightbox.github_issue_label.size: 1 error(s) occurred:

* module.brightbox.github_issue_label.size[2]: github_issue_label.size.2: GET https://api.github.com/repos/terraform-providers/terraform-provider-brightbox/labels/size/M: 502 Server Error []
* module.rancher.github_issue_label.question: 1 error(s) occurred:

* module.rancher.github_issue_label.question: github_issue_label.question: GET https://api.github.com/repos/terraform-providers/terraform-provider-rancher/labels/question: 502 Server Error []
* module.tfe.github_issue_label.regression: 1 error(s) occurred:

* module.tfe.github_issue_label.regression: github_issue_label.regression: GET https://api.github.com/repos/terraform-providers/terraform-provider-tfe/labels/regression: 502 Server Error []

Steps to Reproduce

1. terraform apply

Important Factoids

The Terraform state above has 2600+ github_issue_label resources. (Occurs more often on larger amounts of github_issue_label resources.)

References

• https://github.com/terraform-providers/terraform-provider-github/pull/200 -- while batching could help reduce the occurrence of these, the root problem would remain

[majormoses]

Ya I have seen this a few times as well, adding some very basic retries would be good. We might want to consider putting in some kind of circuit breaker with that many objects as this could easily contribute to being rate limited with a small outage once its over and being left in an odd in between state.

[captn3m0]

The number of failures we've been getting due to these random 502s is steadily increasing. Now every plan takes 3-5 attempts before we get lucky.

[captn3m0]

I reached out to GitHub support, and they mentioned that having the x-github-request-id Header alongside such failures would be helpful. Would be nice if we could add that to terraform error message, in case of failures.

[bflad]

@captn3m0 that sounds like a great enhancement! It should probably be added as a top level feature request so it doesn't get lost in here since they would likely require two different implementations. 👍

[mattmichal]

I recently had a failure due to a 503 error. Due to API limits, we're using read delay and write delay which means that plans can take a long time to run. A single 503 error requires that the plan run again, taking up additional time that is better spent on other things. An HTTP call retry would be a very valuable feature.

[dcfranca]

Funny thing, ChatGPT things it is already implemented:

Yes, Terraform supports retrying operations for the GitHub provider by setting the retryable_errors and max_retries configuration options in the provider block.

Here's an example of how to configure retrying for the GitHub provider in Terraform:

provider "github" {
  token = "YOUR_GITHUB_TOKEN"
  retryable_errors = ["429", "500", "502", "503", "504"]
  max_retries = 5
}

[github-actions]

👋 Hey Friends, this issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Please add the Status: Pinned label if you feel that this issue needs to remain open/active. Thank you for your contributions and help in keeping things tidy!