Terraform init is extremely slow

[AlmogCohen]

Terraform init takes forever and it seems like it is due to provider download. This extremely slow behaviour is consistent for the last 2 days. Working like that is hellish as I'm using terragrunt which runs the terraform init command often in my case.

Drilling down on the trace logs, I've seen that the download of https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip takes usually 1-5 minutes minimum. When I simply download this url in my browser, it never takes more than 15 seconds....usually less than 10. When I run the init command with trace logs, it is almost always takes many many minutes to finish.

I have no internet issues at ALL

Terraform Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.22.0

Terraform Configuration Files

I have created only one file in my example project, backend.tf and this is the its content

terraform {
  backend "s3" {
    encrypt        = true
    key            = "staging/elastic_beanstalk/terraform.tfstate"
    profile        = "some-profile"
    region         = "us-east-1"
    bucket         = "company-terraform-state"
    dynamodb_table = "terraform-lock-table"
  }
}

Debug Output

https://gist.github.com/AlmogCohen/b8af6e33c170c393a63e126226ad7108

Crash Output

Not relevant

Expected Behavior

Terraform provider download should take as fast as it takes in the browser, and terraform init should not take 5 minutes for a very very simple terraform proejct

Actual Behavior

Terraform provider/plugin downloads takes forever

Steps to Reproduce

1. terraform init

Additional Context

I'm using terragrunt, so the init command runs often on one of my dependency projects. It doesn't really matter for this bug report because I've run the terraform command directly in a folder which has only tf files in.

References

Not relevant

[AlmogCohen]

Here is another trace log example from a project that is includes a little more pieces https://gist.github.com/AlmogCohen/19495e839bc4894d8084e9ab191e23f6

A billion of things are being downloaded, but mostly when it is requested from the hashicorp releases host, the download takes forever.

The download of https://releases.hashicorp.com/terraform-provider-aws/2.70.0/terraform-provider-aws_2.70.0_darwin_amd64.zip took almost a minute. Download of https://releases.hashicorp.com/terraform-provider-template/2.2.0/terraform-provider-template_2.2.0_darwin_amd64.zip took almost 40 seconds via the terraform init command, while via browser it took literally 2 seconds.

[danieldreier]

Thanks for reporting this! You're right that it shouldn't be so slow. I'm running on a mac and it's not that slow for me. Can you try and do an install of terraform via homebrew? Homebrew's Terraform has cgo enabled, and I'm wondering if you're stalling on DNS resolution.

[AlmogCohen]

Thanks! I appreciate your quick (and affirming) response! I'm pretty sure I'm using the homebrew version as I've upgraded from Terraform v0.12 a few days ago...the last homebrew command I have in history is `brew reinstall terraform@0.13`. If I had DNS issues...won't it be affecting the browser download speed as well? For all modules I tried, the browser speed was like 20 times faster than the `terraform init` for that specific module. EDIT: I've just verified locally, the terraform init is still extremely slow

[zzn01]

same issue here, terrafrom version is 0.13.5, installed via homebrew
hang on aws provider for very long time(10 minutes+)

- Installing -/aws v3.22.0...

[danieldreier]

@AlmogCohen the reason I wanted to look into DNS issues is that golang uses its own DNS resolver rather than the native one, and this has caused issues in Terraform (and many other projects) that can currently only be worked around with a cgo-enabled build. However, if you're not seeing it on a homebrew build then that's probably not the issue.

Most of the Terraform Core engineering team is out until next week, so I'm just doing basic triage of urgent issues. As other folks run into this - please upvote with a 👍 rather than a comment, to help us prioritize this. I don't think this will get looked at until the team is back but if it turns out there's a more widespread issue I can call folks in.

[danieldreier]

I went and tried to reproduce this again on my mac with the following trivial config:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

My init took about 7 seconds:

time terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terraform init  1.77s user 0.56s system 33% cpu 6.883 total

I do believe you that you're running into this, just wanted to clarify that I'm not able to reproduce a more widespread problem.

Can you share a traceroute to releases.hashicorp.com? Okay to obscure the first few hops for privacy, I'm just trying to get clues on what's different between our environments. Also, can you share what OS version you're running, if it's X86 vs ARM, anything interesting about your network environment like an HTTP proxy, etc?

[AlmogCohen]

I copied your trivial config into main.tf and run time terraform init and this is the result:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terraform init  2.73s user 1.57s system 7% cpu 1:00.33 total

Just to prove the point again regarding no internet issues, I've run time wget https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip and got this:

--2020-12-31 08:19:33--  https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.1.183, 151.101.129.183, 151.101.65.183, ...
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.1.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 42005468 (40M) [application/zip]
Saving to: ‘terraform-provider-aws_3.22.0_darwin_amd64.zip’

terraform-provider-aws_3.22.0_darwin_amd64.zip     100%[================================================================================================================>]  40.06M  10.1MB/s    in 4.7s

2020-12-31 08:19:38 (8.61 MB/s) - ‘terraform-provider-aws_3.22.0_darwin_amd64.zip’ saved [42005468/42005468]

wget   0.10s user 0.25s system 6% cpu 5.137 total

Mac specifications:

For the traceroute, I think some Firewall drops it midway because nothing can be seen after the first few hops:

traceroute -q 1 -w 1 -m 100 releases.hashicorp.com
traceroute: Warning: releases.hashicorp.com has multiple addresses; using 151.101.1.183
traceroute to dualstack.s.shared.global.fastly.net (151.101.1.183), 100 hops max, 52 byte packets
 1   #### Removed #####
 2   #### Removed #####
 3  *
 4   #### Removed #####
 5  #### Removed #####
 6  *
 7  bb302-pt-e49-100g-hot.israelinternet.co.il (185.149.252.109)  11.057 ms
 8  10.141.226.60 (10.141.226.60)  12.209 ms
 9  10.185.180.51 (10.185.180.51)  15.525 ms
10  10.185.180.11 (10.185.180.11)  12.616 ms
11  be6037.203.rcr22.fra06.atlas.cogentco.com (149.14.210.161)  69.101 ms
12  *
13 *
.......(nothing answered here....)
97  *
98  *
99  *
100  *