-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Terraform init is extremely slow #27379
Comments
Here is another trace log example from a project that is includes a little more pieces https://gist.github.com/AlmogCohen/19495e839bc4894d8084e9ab191e23f6 A billion of things are being downloaded, but mostly when it is requested from the hashicorp releases host, the download takes forever. The download of |
Thanks for reporting this! You're right that it shouldn't be so slow. I'm running on a mac and it's not that slow for me. Can you try and do an install of terraform via homebrew? Homebrew's Terraform has cgo enabled, and I'm wondering if you're stalling on DNS resolution. |
Thanks! I appreciate your quick (and affirming) response!
I'm pretty sure I'm using the homebrew version as I've upgraded from Terraform v0.12 a few days ago...the last homebrew command I have in history is `brew reinstall terraform@0.13`.
If I had DNS issues...won't it be affecting the browser download speed as well? For all modules I tried, the browser speed was like 20 times faster than the `terraform init` for that specific module.
EDIT: I've just verified locally, the terraform init is still extremely slow
|
same issue here, terrafrom version is 0.13.5, installed via homebrew
|
@AlmogCohen the reason I wanted to look into DNS issues is that golang uses its own DNS resolver rather than the native one, and this has caused issues in Terraform (and many other projects) that can currently only be worked around with a cgo-enabled build. However, if you're not seeing it on a homebrew build then that's probably not the issue. Most of the Terraform Core engineering team is out until next week, so I'm just doing basic triage of urgent issues. As other folks run into this - please upvote with a 👍 rather than a comment, to help us prioritize this. I don't think this will get looked at until the team is back but if it turns out there's a more widespread issue I can call folks in. |
I went and tried to reproduce this again on my mac with the following trivial config: terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 3.0"
}
}
}
# Configure the AWS Provider
provider "aws" {
region = "us-east-1"
} My init took about 7 seconds:
I do believe you that you're running into this, just wanted to clarify that I'm not able to reproduce a more widespread problem. Can you share a traceroute to releases.hashicorp.com? Okay to obscure the first few hops for privacy, I'm just trying to get clues on what's different between our environments. Also, can you share what OS version you're running, if it's X86 vs ARM, anything interesting about your network environment like an HTTP proxy, etc? |
I copied your trivial config into
Just to prove the point again regarding no internet issues, I've run
For the
|
@AlmogCohen Thanks for running the timings and providing additional data. One suggestion I have if you're running init often (not a workflow that is currently optimized), do make sure that if you can you're using provider cache settings. This should help with regard to your workflow. On the final comments with timing, I see @danieldreier's report with ~7 seconds, and yours with ~5 seconds ... neither of these are close to the original 1-5 minutes in the original report. @danieldreier was also correct to point to CGO builds, as there are known DNS issues/traps that occur with other versions. Would enabling the cache help your workflow? |
@pselle Thanks for your comment. I'll look into enabling the cache later this week and respond here. Regardless, it is important to note that the original problem still stands in place. The timing was 1 minute (and not 5 seconds like you said). If you'll take a closer look, you can see that the 5s timing is of wget. Terraform init is still 12 times slower, and from previous logs you could see that 95% of the time spent on terraform init is due to the resource download of the aws provider (which was timed as 5s via wget). |
@AlmogCohen Thanks, you're correct, I was looking at the wrong timing! It was the 1 minute. |
I have the same problem on Windows. Checking with Wireshark, I can see that the releases.hashicorp.com server starts off by accepting a TLSv1.2 connection but then gets stuck returning messages saying "ignored unknown record". It took so long that I killed it. I was just doing the learn-terraform-aws-instance tutorial. I was able to create infrastructure once, but never again after that. My provider has a short MTU of 1472 bytes. I think this requires servers to accept ICMP messages saying that packets are too long and must be resent with the shorter packet size. If routers are dropping ICMP packets it would be a problem. Just a thought. |
Although learn-terraform-aws-instance tutorial fails completely on my office connection as described above, it works just fine when executed from Windows Server hosted by a different provider. Other Windows PCs in my office also fail, but someone was able to get it to work on Linux. So it's the combination of my office connection + Windows that fails. |
Something that also aggravates this issue is the fact that apparently, if you run multiple consecutive $ terraform init
Initializing modules...
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
...
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/random v3.0.0...
- Installed hashicorp/random v3.0.0 (signed by HashiCorp)
- Installing hashicorp/local v2.0.0...
- Installed hashicorp/local v2.0.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
...
Terraform has been successfully initialized! Enabling cache may improve the speed of this operation, but it will still take a while because in Terraform |
this is a really bad experience when creating modules |
I'm using terraform 0.14.5 on ubuntu. I'm experiencing the same thing.
|
same issue from Singapore on Apple M1 (macOS Monterey 12.2) The provider is as simple as: terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "4.0.0"
}
}
}
provider "aws" {
region = "ap-southeast-1"
} and hang at
|
So is HashiCorp going to comment? |
Having same issue |
I m having the same issue. Any update. Thank you. |
Terraform v1.2.2 ...and soooo sloooooow |
Same issue - trying caching, as recommended above |
Same issue... |
Same issue with Terraform v1.2.5 Interim solution is manual remove |
Had the same issue on my macOS Monterey (12.5). I Set ipv6 to |
only that it is not always possible to change network settings, like in a CI-CD Environment. |
I'm also experiencing this, @jeffery fix seemed to make it faster, but it's still pretty slow to download providers |
I run Terraform via Terragrunt inside a Docker container. Worked perfectly for years, until M2 Macs come around. |
Hi all, It isn't really clear how we can make progress on this issue, because there is no clear signal about what is causing these problems. We've seen several different theories discussed above, including:
As before, we believe you all that Terraform is working more slowly than some other network software when run on your computer but I'm at a loss as to what to do about it unless someone can either describe exactly why Terraform is slow for their particular situation or can describe a situation we can reproduce on our own systems to debug it ourselves. Our team is distributed across the US, Canada, and Europe with different ISPs each, but we've not been able to reproduce these problems on our systems or on the cloud platforms we have access to. If you are able to download provider packages more effectively using software other than Terraform itself, you may benefit from setting up a local mirror of the providers you use regularly on your computer, and then setting up a custom provider installation strategy. That means you can in principle download the provider If anyone does have any evidence of specific network misbehavior in Terraform, please let us know! |
@apparentlymart your thorough response is very much appreciated! For the protocol, as the original issue reporter....years have passed and I've since upgraded my Macbook, changed my address, my ISP and the terraform versions I use (now always >1.0.0 for all projects). I haven't seen the issue I reported for years now... The fact that "it works on my machine" does not invalidate the many other reports that came in and seem to be related to the very same "stage" of the terraform init. On the other hand, I do understand how challenging it may be to take any action here to resolve the (many?) issues. |
I try to install terraform providers locally on my machine (Linux, Manjaro). Installing them is currently really slow (I usually do not use or at least initialize terraform locally, but run it from a CI, where some providers also might be cached). I notice that slow performance only with terraform. E.g. pulling Docker images from Docker Hub is fast as usual. I grabbed a sample provider plugin from the terraform logs and tried to download it via chrome. Here are the screenshots:
Location: Germany |
Same issue. I guess #32863 is a duplicate of this. |
I think terraform init is slow even when its working! |
Below command took over 46 minutes. The slow part was Using MacOS. The only network related thing that is slow on this machine is Terraform.
|
Seems that https://releases.hashicorp.com/ is some times extremely slow for downloading. Download speeds are e.g. 34,4KB/s. This is with
|
From my experience, only the first time of init takes time to download (cache) the provider binary. As long as the binary got downloaded and cached to local environment, init will take effect instantly. One solution might be: Download and unzip all the provider binary(ies) to your local terraform provider registry. Reference URL: https://releases.hashicorp.com Most dev providers can be downloaded from their github repositories. |
Perhaps theres something I'm not getting. run locally it takes about 8secs at the moment, and reuses the lock file I have committed. I still don't think thats fast for my basic project, doesn't seem like good caching behaviour to me.
and my circle ci runner + terraform cloud runs always have to do more than that too even with the updated lock file:
|
Ok I took a look at fixing this - it was due to my caching not setup properly. but still takes about 6 seconds here when on my mac (speed test is about 108MBit/s from Melbourne Australia): init in cicrcle ci is now 1-2 secs though with caching fixed so that is nice to know. |
For me, Terraform is consistently slow for me, it's a nightmare (and it's the only slow one on my machine, so definitely a fault of terraform, not my network or machine config). |
Terraform init takes forever and it seems like it is due to provider download. This extremely slow behaviour is consistent for the last 2 days. Working like that is hellish as I'm using terragrunt which runs the
terraform init
command often in my case.Drilling down on the trace logs, I've seen that the download of https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip takes usually 1-5 minutes minimum. When I simply download this url in my browser, it never takes more than 15 seconds....usually less than 10. When I run the
init
command with trace logs, it is almost always takes many many minutes to finish.I have no internet issues at ALL
Terraform Version
Terraform Configuration Files
I have created only one file in my example project,
backend.tf
and this is the its contentDebug Output
https://gist.github.com/AlmogCohen/b8af6e33c170c393a63e126226ad7108
Crash Output
Not relevant
Expected Behavior
Terraform provider download should take as fast as it takes in the browser, and terraform init should not take 5 minutes for a very very simple terraform proejct
Actual Behavior
Terraform provider/plugin downloads takes forever
Steps to Reproduce
terraform init
Additional Context
I'm using terragrunt, so the
init
command runs often on one of my dependency projects. It doesn't really matter for this bug report because I've run theterraform
command directly in a folder which has onlytf
files in.References
Not relevant
The text was updated successfully, but these errors were encountered: