Run "Terraform plan" getting error "Failed to request discovery document"

[jennihan]

Terraform Version

Terraform v0.11.14

Terraform Configuration Files

terraform {
  backend "remote" {
    hostname = "terraform.abc.io"
    organization = "Security"

    workspaces {
      name = "monitoring"
    }
  }
}

Debug Output

Crash Output

Expected Behavior

display plan

Actual Behavior

getting below error

Error configuring the backend "remote": Failed to request discovery document: Get https://terraform.abc.io/.well-known/terraform.json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Please update the configuration in your Terraform files to fix this error
then run this command again.

Steps to Reproduce

1. terraform init
2. terraform plan

Additional Context

References

[teamterraform]

Hi @jennihan!

Based on the fact that you are using the remote backend with a hostname other than app.terraform.io, we're assuming you're a Terraform Enterprise customer. If so, we'd recommend contacting the Terraform Enterprise support team about this, as they can provide more tailored advice due to having more context about how your Terraform Enterprise deployment is configured.

[FortuneLenovo]

@jennihan was the issue has been resolved?

[richardj-bsquare]

I get this at random times also, I don't know if it is a coincidence, but I seem to get it more after an aborted terraform, when I have to manually release the lock... First time it fails with the above and succeeds there-on.

Error: Failed to request discovery document: Get https://app.terraform.io/.well-known/terraform.json: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[skolsuper]

I am getting this a lot recently, seems like a very short timeout is set on this request. Right now in Singapore it's failing for me on ~50% of import, plan or apply commands, seemingly at random.

edit: terraform v0.12.24

[alen-z]

I'm good if I don't use -parallelism.

[danieldreier]

@alen-z can you please also note what terraform version you're using?

[alen-z]

Terraform v0.12.24

[hskiba]

I am getting this a lot recently, seems like a very short timeout is set on this request. Right now in Singapore it's failing for me on ~50% of import, plan or apply commands, seemingly at random.

edit: terraform v0.12.24

I'm getting this intermittently running terraform apply from Malaysia (usually in the evenings)...

Terraform v0.12.26

[skayle-denis]

Same intermittent error with 0.12.28 from Sydney

[fethi16]

I Have the same problem:
terraform 0.12.24

---

2020/06/29 09:09:29 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/06/29 09:09:32 [ERR] Checkpoint error: Get https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=df7afca7-9552-b80b-2298-4a8246bd841f&version=0.12.24: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020/06/29 09:09:39 [DEBUG] Failed to request discovery document: Get https://registry.terraform.io/.well-known/terraform.json: proxyconnect tcp: net/http: TLS handshake timeout

@jim-hashicorp
@hashicorp-cloud

[levelfivehub]

Same problem here - London, UK

[ghost]

Having the same problem lately. So annoying. My continuous integration is blocked.

[jacoor]

I started having this today.
Poland, Germany (VPN), USA (VPN), second network... all fails.

[skolsuper]

What's annoying about this issue is that it will probably never affect anyone with commit access.

[jacoor]

In my case, it occurred to be some weird issue with aws-vault and credentials.
I used aws-vault with --no-session parameter and it started to fail. After opening the new terminal window problem disappeared.

[Vivalldi]

If you're a mac user you can try removing expired certificates from your keychain - https://discuss.hashicorp.com/t/error-when-running-terraform-init/3135/3

Make sure you go through both system & login certificates

[zzxwill]

I also hit this issue in an alphine image. It failed due to Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=6935a4be-5f95-e966-58da-a1fe8d135790&version=0.14.10".

bash-5.1# export TF_LOG=TRACE
bash-5.1# terraform init
2021/06/29 08:55:37 [INFO] Terraform version: 0.14.10
2021/06/29 08:55:37 [INFO] Go runtime version: go1.15.6
2021/06/29 08:55:37 [INFO] CLI args: []string{"/usr/bin/terraform", "init"}
2021/06/29 08:55:37 [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2021/06/29 08:55:37 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021/06/29 08:55:37 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021/06/29 08:55:37 [DEBUG] ignoring non-existing provider search directory /root/.terraform.d/plugins
2021/06/29 08:55:37 [DEBUG] ignoring non-existing provider search directory /root/.local/share/terraform/plugins
2021/06/29 08:55:37 [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2021/06/29 08:55:37 [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2021/06/29 08:55:37 [INFO] CLI command args: []string{"init"}
2021/06/29 08:55:37 [TRACE] ModuleInstaller: installing child modules for . into .terraform/modules
2021/06/29 08:55:37 [DEBUG] Module installer: begin vpc
2021/06/29 08:55:37 [TRACE] ModuleInstaller: vpc is not yet installed
2021/06/29 08:55:37 [TRACE] ModuleInstaller: cleaning directory .terraform/modules/vpc prior to install of vpc
2021/06/29 08:55:37 [TRACE] ModuleInstaller: vpc is a registry module at terraform-google-modules/network/google
2021/06/29 08:55:37 [DEBUG] vpc listing available versions of terraform-google-modules/network/google at registry.terraform.io
2021/06/29 08:55:37 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2021/06/29 08:55:37 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
Initializing modules...
2021/06/29 08:55:40 [ERR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=6935a4be-5f95-e966-58da-a1fe8d135790&version=0.14.10": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/06/29 08:55:48 [TRACE] modsdir: writing modules manifest to .terraform/modules/modules.json

Error: Error accessing remote module registry

Failed to retrieve available versions for module "vpc" (main.tf:1) from
registry.terraform.io: Failed to request discovery document: Get
"https://registry.terraform.io/.well-known/terraform.json": context deadline
exceeded (Client.Timeout exceeded while awaiting headers).

[ashuraits]

+1 Facing this issue every day in Terraform cloud apply

[pbmanju]

+1 recently I started using terraform cloud as remote backend and boom hitting this issue on regular basis.

Initializing the backend...
│ Error: Failed to request discovery document: Get "https://app.terraform.io/.well-known/terraform.json": net/http: request canceled (Client.Timeout exceeded while awaiting headers)

[sergei-ivanov]

It started being an issue for me on Saturday 11-Sep-2021.

I am not using any backend, in fact I am running terraform init -backend=false on something like 200 local terraform modules in order to run tflint and other tools on them later.

$ terraform version
Terraform v1.0.6
on linux_amd64

The error I am getting is related to downloading a module:

Initializing modules...
- label in ../label
╷
│ Error: Error accessing remote module registry
│ 
│ Failed to retrieve available versions for module "label" (../label/main.tf:1) from registry.terraform.io: Failed to
│ request discovery document: Get "https://registry.terraform.io/.well-known/terraform.json": net/http: request
│ canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
╵

╷
│ Error: Error accessing remote module registry
│ 
│ Failed to retrieve available versions for module "label" (../label/main.tf:1) from registry.terraform.io: Failed to
│ request discovery document: Get "https://registry.terraform.io/.well-known/terraform.json": net/http: request
│ canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
╵

The module in question is:

module "label" {
  source             = "cloudposse/label/null"
  version            = "0.24.1"
  # ...
}

Occasionally it also fails on downloading provider information (even though I use lock files and local caching of providers):

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/aws: could not query provider registry for
│ registry.terraform.io/hashicorp/aws: the request failed after 2 attempts, please try again later: Get
│ "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for
│ connection (Client.Timeout exceeded while awaiting headers)
╵

I also noticed that registry.terraform.io is dualstack and previously requests to IPv6 were failing due to local misconfiguration of IPv6. But that has been fully rectified now, and using curl I can see that both IPv4 and IPv6 work.

I am in the UK on a 300Mbps fibre broadband from BT, so unless it's a BT issue (unlikely in this case), it must be something on HashiCorp end. I wonder if there's some API rate limiting or throttling in place, but I could not find anything to that effect in the documentation. I have also checked incident status on HashiCorp site, and all's green.

[sergei-ivanov]

I started a loop to curl terraform registry every second, and I started Terraform init for multiple modules in parallel.

while true; do curl -v https://registry.terraform.io/.well-known/terraform.json ; sleep 1; done

At some point both choked. Terraform bailed out pretty quickly, curl was stuck for longer but also gave up in the end. Here's the output from curl:

*   Trying 2a04:4e42:4::561:443...
* TCP_NODELAY set
*   Trying 151.101.18.49:443...
* TCP_NODELAY set
* connect to 2a04:4e42:4::561 port 443 failed: Connection timed out
* After 85412ms connect time, move on!
* connect to 151.101.18.49 port 443 failed: Connection timed out
* Failed to connect to registry.terraform.io port 443: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to registry.terraform.io port 443: Connection timed out

After that the curl loop carried on as normal.

I am still not sure if it's a local issue, an issue with my provider (BT), an issue with Terraform registry, or an issue with HashiCorp CDN (Fastly). I'm not sure what else I can do to help diagnose this problem.

[sergei-ivanov]

OK, I think I have isolated and resolved the issue in my case. It's always DNS to blame in the end, right? I hardcoded CloudFlare DNS (1.1.1.1 and its IPv4 and IPv6 aliases) into my network settings on the laptop, and since then everything seems to be working like a treat.

Looks like there must have been a rogue DNS server on BT side that was serving stale DNS records for registry.terraform.io or Fastly, and occasionally they were picked up locally due to relatively short TTL. I rebooted both the laptop and the router earlier, so I am pretty sure the problem was not on my end.

I hope maybe this helps someone on this thread.

[camin007]

It can also be that only the secure connection is allowed that is HTTPS whereas the backend is making HTTP calls.

[iamolegga]

This issue is still actual on macos with the latest terraform release:

╷
│ Error: Error getting client
│
│   with provider["registry.terraform.io/hashicorp/tfe"],
│   on providers.tf line 7, in provider "tfe":
│    7: provider "tfe" {}
│
│ Error getting client: Failed to request discovery document: Get "https://app.terraform.io/.well-known/terraform.json": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
╵
Releasing state lock. This may take a few moments...
λ  terraform version
Terraform v1.4.0-alpha20221109

@Vivalldi mentioned certificates and here it's mentioned too as a bug of golang.

But it's solved in golang/go@feb024f which is included in next releases: go1.19.3 go1.19.2 go1.19.1 go1.19 go1.19rc2 go1.19rc1 go1.19beta1 go1.18.8 go1.18.7 go1.18.6 go1.18.5 go1.18.4 go1.18.3 go1.18.2 go1.18.1 go1.18 go1.18rc1 go1.18beta2 go1.18beta1

terraform is using go1.18 since v1.4.0-alpha20221109 v1.3.5 v1.3.4 v1.3.3 v1.3.2 v1.3.1 v1.3.0 v1.3.0-rc1 v1.3.0-beta1 v1.3.0-alpha20220817 v1.3.0-alpha20220803 v1.3.0-alpha20220706 v1.3.0-alpha20220622

So since v1.3.0-alpha20220622 this is surely not a bug of golang.

Maybe someone from the team could provide more info on this and when this annoying bug will be fixed. The only solution I can see now is move from terraform cloud to another backend

[iamolegga]

@danieldreier any chance this will be fixed soon? Sorry for bothering, but there are no any comments from the maintainers for 2.5 years

[danieldreier]

@iamolegga I’m no longer working at HashiCorp, so I can’t help get it prioritized

[crw]

@iamolegga is this happening with the remote backend or the cloud backend? https://developer.hashicorp.com/terraform/cli/cloud/settings

[iamolegga]

@crw in my case it's cloud

terraform {
  cloud {
    organization = "my-org"
    workspaces {
      name = "my-workspace"
    }
  }

  required_providers { ... }
}

[crw]

Hi @iamolegga, do you have any firewall software on your system, and it is configured to allow this access? Also, could you please link us to the debug trace output? The following is excerpted from our new issue form, which explains how to prepare a debug log:

Debug Output
Full debug output can be obtained by running Terraform with the environment variable TF_LOG=trace. Please create a GitHub Gist containing the debug output. Please do not paste the debug output in the issue, since debug output is long. Debug output may contain sensitive information. Please review it before posting publicly.

Thanks! cc: @annawinkler

[crw]

@iamolegga, as this is an issue with Terraform Cloud, you might also considered opening this through the official support channels. If you choose to do so, please email tf-cloud@hashicorp.support or open a new request.

[annawinkler]

👋 Just wanted to pop in to say that we're looking into this issue. Hopefully I'll have an update this week with more. ✨

[annawinkler]

We addressed two areas as part of this issue:

1. We've improved the performance of the endpoint that returns the discovery document in TFC. This improvement is available now! Hopefully with better performance we'll have fewer overall discovery document request errors.
2. In the next terraform release (1.4), if you do run into this problem, the error message will have more context indicating that it's most likely a network issue.

Closing out the issue now. If folks still see problems, let's open up a new issue.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.