Terrafrom Crash

[ghost]

This issue was originally opened by @cah-ed-hammond as hashicorp/terraform#25742. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

This works:

Terraform v0.12.29
+ provider.google v3.23.0
+ provider.null v2.1.2

This causes the crash:

Terraform v0.12.29
+ provider.google v3.32.0              << note latest release (downloaded to clean directory)
+ provider.null v2.1.2

Terraform Configuration Files

Debug Output

Error: rpc error: code = Unavailable desc = transport is closing

panic: nil context
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe:
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: goroutine 16 [running]:
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: net/http.(*Request).WithContext(...)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/goenv/versions/1.14.5/src/net/http/request.go:358
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/net/context/ctxhttp.Do(0x0, 0x0, 0x3bbe6a0, 0xc000136700, 0x788c769d3bebe7f2, 0x10, 0x219f400)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/net@v0.0.0-20200324143707-d3edc9973b7e/context/ctxhttp/ctxhttp.go:27 +0x32b
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/oauth2/internal.doTokenRoundTrip(0x0, 0x0, 0xc000136700, 0x26, 0xc000035ae0, 0x18)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/oauth2@v0.0.0-20200107190931-bf48bf16ab8d/internal/token.go:232 +0x82
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/oauth2/internal.RetrieveToken(0x0, 0x0, 0xc00018aff0, 0x26, 0xc000035ae0, 0x18, 0x27c560d, 0x23, 0xc0001b32a0, 0x1, ...)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/oauth2@v0.0.0-20200107190931-bf48bf16ab8d/internal/token.go:202 +0xe6
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/oauth2.retrieveToken(0x0, 0x0, 0xc0006fe0e0, 0xc0001b32a0, 0xc0001b3370, 0x1a, 0x1)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/oauth2@v0.0.0-20200107190931-bf48bf16ab8d/token.go:157 +0x99
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/oauth2.(*tokenRefresher).Token(0xc0006f7170, 0x4adbf4, 0x27864c1, 0x6)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/oauth2@v0.0.0-20200107190931-bf48bf16ab8d/oauth2.go:270 +0x1e5
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/oauth2.(*reuseTokenSource).Token(0xc0007cb380, 0x0, 0x0, 0x0)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/oauth2@v0.0.0-20200107190931-bf48bf16ab8d/oauth2.go:304 +0xaf
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: golang.org/x/oauth2.(*Transport).RoundTrip(0xc0007cb3c0, 0xc000136600, 0x0, 0x0, 0x0)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/golang.org/x/oauth2@v0.0.0-20200107190931-bf48bf16ab8d/transport.go:45 +0xa5
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/helper/logging.(*transport).RoundTrip(0xc0007cb3e0, 0xc000136600, 0xc0001b3610, 0x1, 0x1)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/logging/transport.go:27 +0x16b
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: github.com/terraform-providers/terraform-provider-google/google.(*retryTransport).RoundTrip(0xc0006f7230, 0xc000136400, 0x0, 0x0, 0x0)
2020-08-04T10:41:42.035-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/google/retry_transport.go:125 +0x2cc
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: net/http.send(0xc000136100, 0x2c36ea0, 0xc0006f7230, 0xbfc27ad101e35854, 0x7138733d1, 0x3bbe9e0, 0xc000a20e18, 0xbfc27ad101e35854, 0x1, 0x0)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/goenv/versions/1.14.5/src/net/http/client.go:252 +0x445
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: net/http.(*Client).send(0xc0006f7200, 0xc000136100, 0xbfc27ad101e35854, 0x7138733d1, 0x3bbe9e0, 0xc000a20e18, 0x0, 0x1, 0x2030001)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/goenv/versions/1.14.5/src/net/http/client.go:176 +0x101
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: net/http.(*Client).do(0xc0006f7200, 0xc000136100, 0x0, 0x0, 0x0)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/goenv/versions/1.14.5/src/net/http/client.go:699 +0x451
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: net/http.(*Client).Do(...)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/goenv/versions/1.14.5/src/net/http/client.go:567
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: google.golang.org/api/internal/gensupport.SendRequest(0x0, 0x0, 0xc0006f7200, 0xc000136100, 0xc000086540, 0x59, 0x2c36220)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/api@v0.26.0/internal/gensupport/send.go:42 +0x244
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: google.golang.org/api/storage/v1.(*BucketsInsertCall).doRequest(0xc0001b3d50, 0x2784c0d, 0x4, 0x0, 0xb88413a501d9cb01, 0xc000342160)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/api@v0.26.0/storage/v1/storage-gen.go:4076 +0x77d
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: google.golang.org/api/storage/v1.(*BucketsInsertCall).Do(0xc0009a3d50, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/api@v0.26.0/storage/v1/storage-gen.go:4088 +0xa8
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: github.com/terraform-providers/terraform-provider-google/google.resourceStorageBucketCreate.func1(0xc0002079c8, 0xc0009a3ea4)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/google/resource_storage_bucket.go:429 +0x185
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: github.com/terraform-providers/terraform-provider-google/google.retryTimeDuration.func1(0x495997)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-google/google/retry_utils.go:21 +0x39
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/helper/resource.Retry.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/resource/wait.go:22 +0x65
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: github.com/hashicorp/terraform-plugin-sdk/helper/resource.(*StateChangeConf).WaitForState.func1(0xc000207860, 0xc0006fed20, 0xc00069a840, 0xc000086120, 0xc0009df380, 0xc0009df378)
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/resource/state.go:103 +0x29f
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: created by github.com/hashicorp/terraform-plugin-sdk/helper/resource.(*StateChangeConf).WaitForState
2020-08-04T10:41:42.036-0400 [DEBUG] plugin.terraform-provider-google_v3.32.0_x5.exe: /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk@v1.11.0/helper/resource/state.go:80 +0x1c6
2020/08/04 10:41:42 [DEBUG] google_storage_bucket.thisBucket: apply errored, but we're indicating that via the Error pointer rather than returning it: rpc error: code = Unavailable desc = transport is closing
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalMaybeTainted
2020/08/04 10:41:42 [TRACE] EvalMaybeTainted: google_storage_bucket.thisBucket encountered an error during creation, so it is now marked as tainted
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalWriteState
2020/08/04 10:41:42 [TRACE] EvalWriteState: removing state object for google_storage_bucket.thisBucket
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalApplyProvisioners
2020/08/04 10:41:42 [TRACE] EvalApplyProvisioners: google_storage_bucket.thisBucket has no state, so skipping provisioners
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalMaybeTainted
2020/08/04 10:41:42 [TRACE] EvalMaybeTainted: google_storage_bucket.thisBucket encountered an error during creation, so it is now marked as tainted
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalWriteState
2020/08/04 10:41:42 [TRACE] EvalWriteState: removing state object for google_storage_bucket.thisBucket
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalIf
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalIf
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalWriteDiff
2020/08/04 10:41:42 [TRACE] : eval: *terraform.EvalApplyPost
2020/08/04 10:41:42 [ERROR] : eval: *terraform.EvalApplyPost, err: rpc error: code = Unavailable desc = transport is closing
2020/08/04 10:41:42 [ERROR] : eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/08/04 10:41:42 [TRACE] [walkApply] Exiting eval tree: google_storage_bucket.thisBucket
2020/08/04 10:41:42 [TRACE] vertex "google_storage_bucket.thisBucket": visit complete
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "output.bucket_name" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "google_service_account.thisSA" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "output.service_account_name" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "google_storage_bucket_iam_member.thisBucket-thisSA" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "provider.google (close)" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "null_resource.createJsonKey" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "provider.null (close)" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "provisioner.local-exec (close)" errored, so skipping
2020/08/04 10:41:42 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: reading latest snapshot from terraform.tfstate
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: read nil snapshot
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: no original state snapshot to back up
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: state has changed since last snapshot, so incrementing serial to 1
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: writing snapshot at terraform.tfstate
2020-08-04T10:41:42.054-0400 [DEBUG] plugin: plugin process exited: path=C:\Users\XXXXXXXXXXXXX\AppData\Local\Temp\tmpz.terraform\plugins\windows_amd64\terraform-provider-google_v3.32.0_x5.exe pid=10856 error="exit status 2"
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info
2020/08/04 10:41:42 [TRACE] statemgr.Filesystem: unlocked by closing terraform.tfstate
2020-08-04T10:41:42.058-0400 [DEBUG] plugin: plugin exited
2020-08-04T10:41:42.059-0400 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-08-04T10:41:42.074-0400 [DEBUG] plugin: plugin process exited: path=C:\Users\XXXXXXXXXXXXX\AppData\Local\Temp\tmpz.terraform\plugins\windows_amd64\terraform-provider-null_v2.1.2_x4.exe pid=18076
2020-08-04T10:41:42.074-0400 [DEBUG] plugin: plugin exited
2020-08-04T10:41:42.097-0400 [DEBUG] plugin: plugin process exited: path=C:\Users\XXXXXXXXXXXXXXXXX\bin\terraform.exe pid=21716
2020-08-04T10:41:42.097-0400 [DEBUG] plugin: plugin exited

Crash Output

https://gist.github.com/cah-ed-hammond/dde36b23ac382ddd2db6ba57d568d2e3

Expected Behavior

Create resources as I was able to three weeks ago.
It works with Google plugin v3.23.0 while v3.32.0 crashes.

Actual Behavior

See crash Gist

Steps to Reproduce

Started with clean directory, downloaded latest Terraform exe, downloaded latest Google plugin.
terraform init and terraform plan work fine. terraform apply crashes whey "yes" is entered to proceed.

Additional Context

References

[nat-henderson]

This is a duplicate of #6877, was most likely fixed by GoogleCloudPlatform/magic-modules#3800, which will be released in 3.34.0, next monday. This issue seems to be intermittent and environment-dependent, and we haven't been able to reproduce it, unfortunately.

[nat-henderson]

I'm going to close this in favor of tracking at #6877.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!