Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incident summary for Terraform Registry module download failures (affects TF 0.12.0 to 0.12.18) #23834

Closed
justincampbell opened this issue Jan 10, 2020 · 9 comments

Comments

@justincampbell
Copy link
Member

@justincampbell justincampbell commented Jan 10, 2020

Terraform 0.12 contained a bug with go-getter 1.3.0, where HEAD requests which did not return a Content-Length header in the response were not handled properly. This was fixed in go-getter 1.4.1 and Terraform 0.12.19. The fix is detailed in hashicorp/go-getter#227.

The Terraform Registry uses GitHub for both user authentication and module publishing/hosting. The go-getter issue was discovered because in mid-December 2019, GitHub began omitting the Content-Length header from HEAD requests for some percentage of their traffic. On January 9th, 2020, it became apparent that all requests now exhibit this behavior, and that all Terraform Registry module downloads for these Terraform versions were affected.

Today (January 10th, 2020), we deployed a change to the Terraform Registry which detects requests from the affected Terraform versions and returns a Git go-getter URL (as opposed to HTTPS), which should allow these versions to successfully install modules from the Terraform Registry.

Although we believe this workaround will solve the issue for most users, we recommend that all Terraform 0.12 users upgrade to the latest version (TF 0.12.19 or later). Thanks!

@justincampbell justincampbell pinned this issue Jan 10, 2020
@pwelch pwelch changed the title Incident summary for Terraform Registry module download failures (affects TF 0.12.0 to 0.12.18 Incident summary for Terraform Registry module download failures (affects TF 0.12.0 to 0.12.18) Jan 10, 2020
@kxavier-ims

This comment has been minimized.

Copy link

@kxavier-ims kxavier-ims commented Jan 10, 2020

Hi,

What about terraform 0.11.x? We are experiencing the same problems downloading the modules and we can't upgrade to 0.12 in the short term.
The error we are getting is "Error downloading modules: Error loading modules: subdir "*" not found"

@justincampbell

This comment has been minimized.

Copy link
Member Author

@justincampbell justincampbell commented Jan 10, 2020

@kxavier-ims Our understanding was that Terraform 0.11.x was not affected, because it contains a version of go-getter prior to the introduction of this bug. Could you open an issue with the specific version you're using, and the output of TF_LOG=1 terraform init? Thanks!

@wershlak

This comment has been minimized.

Copy link

@wershlak wershlak commented Jan 10, 2020

This is not fixed for us with 0.12.12. I don't see any change to the behavior or the url from yesterday. Are you sure the change is deployed?

We're currently migrating from tf 11 to 12 and can't upgrade to 0.12.19 until that is complete.

@gkowalski-google

This comment has been minimized.

Copy link

@gkowalski-google gkowalski-google commented Jan 10, 2020

This is not fixed for us with 0.12.12. I don't see any change to the behavior or the url from yesterday. Are you sure the change is deployed?

We're currently migrating from tf 11 to 12 and can't upgrade to 0.12.19 until that is complete.

We are seeing the same issue. When I am testing locally on a mac with 0.12.6, I have not run into this issue at all. However, our test pipeline running Google Cloud Build with 0.12.6 reproduces the error every time. Thanks for trying to address this!

@justincampbell

This comment has been minimized.

Copy link
Member Author

@justincampbell justincampbell commented Jan 10, 2020

@wershlak @gkowalski-google We've identified that we may not be taking the Terraform version into account when caching this response. We're looking into it now.

@justincampbell

This comment has been minimized.

Copy link
Member Author

@justincampbell justincampbell commented Jan 10, 2020

@wershlak @gkowalski-google Could you try now and let me know if it works? Thanks!

@gkowalski-google

This comment has been minimized.

Copy link

@gkowalski-google gkowalski-google commented Jan 10, 2020

@justincampbell Re-ran one test and we did not get the error! I am going to run some more tests but so far it's looking promising. Thank you so much!

@wershlak

This comment has been minimized.

Copy link

@wershlak wershlak commented Jan 10, 2020

Looks like downloads are working now. Thanks for reopening this and getting it fixed @justincampbell

@kxavier-ims

This comment has been minimized.

Copy link

@kxavier-ims kxavier-ims commented Jan 13, 2020

@kxavier-ims Our understanding was that Terraform 0.11.x was not affected, because it contains a version of go-getter prior to the introduction of this bug. Could you open an issue with the specific version you're using, and the output of TF_LOG=1 terraform init? Thanks!

Hi @justincampbell . We were using a patched version of 0.11.x. Now it is working. Thanks.

@allthingsclowd allthingsclowd unpinned this issue Jan 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.