Feature request: Option to ignore failures for data source "http"

[pwillis-els]

Current Terraform Version

$ terraform version
cliv: Executing /home/willis2/.cliv/terraform=1.1.2/bin/terraform
Terraform v1.1.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.70.0
+ provider registry.terraform.io/hashicorp/http v2.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0

Use-cases

We use the http data source to query a list of current IP CIDRs from vendors such as CloudFlare, so we can add those ranges to our security groups. However, occasionally one of those http sources may return a 500 error. We don't need to pull this value on every single terraform apply, because the old values that were already applied will continue to work until the service returns.

Attempted Solutions

# CloudFlare China IPs
data "http" "cloudflare_china_ips" {
  url = "https://api.cloudflare.com/client/v4/ips?china_colo=1"
  request_headers = {
    Accept = "application/json"
  }
}

This is an example of our code. I have not found a workaround for this issue because Terraform simply dies with "500 Error".

An alternative to this would be to use a null resources with "local-exec" provisioner to call curl or wget, but that would not necessarily be portable (Dockerized versions of Terraform might not include curl, and behavior would have to change on Windows). Since there is already a data source for doing an HTTP GET. we might as well extend its functionality to support ignoring these failures.

(On earlier versions of Terraform, it actually gives no indication that this module was causing the error (even with TF_DEBUG=TRACE); I had to upgrade Terraform to figure out it was coming from this module block.)

Proposal

Support an input to the module such as ignore_errors = true.

Alternate proposal: support specifying the range of return statuses that signify success (http_status_success_range = [ 200, 599 ]).

It also would be great to support passing this as an environment variable so when the error occurs we can temporarily enable/disable this behavior and continue to apply our infrastructure, rather than having to modify the code to ignore errors, which is not necessarily what we want.

References

[DonBower]

We have to be able to not fail on error here, for example if our firewall rules block the site or some other network problem.
I suggest a second output besides .body.
Provide another output .returncode which will return 200, 404, etc.

[theipster]

Agreed with the value of returning the HTTP response status code verbatim and without prematurely acting on it - I've dropped a comment in #114 (comment) - WDYT?

That said, what would you expect to happen if a lower-level error occurred (e.g. DNS resolution failure) which effectively resulted in no HTTP status code being available to evaluate at all? It sounds like you'd still prefer to ignore those types of errors too - in which case, we're probably looking at more generic error handling like try(data.http.foo_request.body, "some fallback body") instead.

[pwillis-els]

The primary functionality needed is to be able to provide an option/mechanism to prevent http data source failure from stopping the whole terraform plan/apply. However you can achieve that, the use case is fulfilled. If try() works, I'm good with that.

The secondary functionality needed would be to provide a cached value, so that a failure does not then remove the firewall rules, and then a subsequent run re-adds the firewall rules.... this would cause some very unusual network issues that would be difficult to track down. It sounds like try() with some data source lookup and some funky function calls could get that done.

Although I'm not sure if older versions of Terraform support try ? There's still a lot of TF code out there on older versions, which is why some sort of option in the provider itself would be great. The provider could take an option like empty_on_error, and return null or an empty string on error, which the Terraform code could check for, and decide that empty means "error" and to do something else. It's messy but simple. An alternative attribute .error would allow better error handling code. (Or some better idea)

A .returncode sounds very useful, but isn't a requirement of this feature request.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.