Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Google provider latest version 4.42.0 is crashing #12917

Assignees

Comments

@msahu-onramp
Copy link

msahu-onramp commented Oct 31, 2022

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Affected Resource(s)

  • google_4.42.0

Terraform Configuration Files

# Copy-paste your Terraform configurations here.
#
# For large Terraform configs, please use a service like Dropbox and share a link to the ZIP file.
# For security, you can also encrypt the files using our GPG public key:
#    https://www.hashicorp.com/security
#
# If reproducing the bug involves modifying the config file (e.g., apply a config,
# change a value, apply the config again, see the bug), then please include both:
# * the version of the config before the change, and
# * the version of the config after the change.

Debug Output

Panic Output

Stack trace from the terraform-provider-google_v4.42.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x24dcb65]

goroutine 1154 [running]:
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycleRuleCondition(0xc000c03340)
github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1083 +0x45
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycle(0xc00159b400)
github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1067 +0x276
github.com/hashicorp/terraform-provider-google/google.setStorageBucket(0x2d59a8b?, 0x44?, 0xc0004b9dc0, {0xc0007c2cc0, 0x1c}, {0xc0003c6150, 0x65})
github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1493 +0xf51
github.com/hashicorp/terraform-provider-google/google.resourceStorageBucketRead(0x7ff5b8a53d58?, {0x2cccd40?, 0xc000147000})
github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:732 +0x3a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x3232400?, {0x3232400?, 0xc00111efc0?}, 0xd?, {0x2cccd40?, 0xc000147000?})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:712 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000c9b0a0, {0x3232400, 0xc00111efc0}, 0xc000718820, {0x2cccd40, 0xc000147000})
github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:1015 +0x585
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000cba3c0, {0x3232400?, 0xc00111ef00?}, 0xc000566580)
github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/grpc_provider.go:613 +0x497
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc00045e320, {0x3232400?, 0xc00111e8d0?}, 0xc0008539e0)
github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/tf5server/server.go:747 +0x41e
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x2c5f7e0?, 0xc00045e320}, {0x3232400, 0xc00111e8d0}, 0xc000df90a0, 0x0)
github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000001e0, {0x32377e8, 0xc00070a680}, 0xc000e6ab40, 0xc000cb2de0, 0x4387e30, 0x0)
google.golang.org/grpc@v1.50.1/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x32377e8, 0xc00070a680}, 0xc000e6ab40, 0x0)
google.golang.org/grpc@v1.50.1/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
google.golang.org/grpc@v1.50.1/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
google.golang.org/grpc@v1.50.1/server.go:963 +0x28a

Error: The terraform-provider-google_v4.42.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Expected Behavior

Actual Behavior

Steps to Reproduce

  1. terraform apply

Important Factoids

References

  • #0000
@philip-harvey
Copy link

The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.

Stack trace from the terraform-provider-google_v4.42.0_x5.exe plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x2ecb605]

@shuyama1
Copy link
Collaborator

@msahu-onramp Thanks for filing the issue. Would you mind sharing your config and debug logs so that I can take a look and reproduce the error? Thanks!

@joe-a-t
Copy link

joe-a-t commented Oct 31, 2022

We are seeing this issue too in all kinds of root modules, pretty much everywhere we are using 4.42.0 is crashing right now. We are trying to reproduce as something sharable

@Zamyatin
Copy link

Zamyatin commented Oct 31, 2022

Can confirm same error with cascading The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled. errors

The plugin encountered an error, and failed to respond to the
plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
details.

Stack Trace:

Stack trace from the terraform-provider-google_v4.42.0_x5 plugin:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x24dcb65]

goroutine 303 [running]:
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycleRuleCondition(0xc0005ac620)
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1083 +0x45
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycle(0xc0006aa0a0)
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1067 +0x276
github.com/hashicorp/terraform-provider-google/google.setStorageBucket(0x2d59a8b?, 0x45?, 0xc00054f6c0, {0xc000316860, 0x1d}, {0xc000baa1c0, 0x65})
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1493 +0xf51
github.com/hashicorp/terraform-provider-google/google.resourceStorageBucketRead(0x7f2fd8150fe8?, {0x2cccd40?, 0xc000e2b000})
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:732 +0x3a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x3232400?, {0x3232400?, 0xc001e3a810?}, 0xd?, {0x2cccd40?, 0xc000e2b000?})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:712 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000d470a0, {0x3232400, 0xc001e3a810}, 0xc000d69790, {0x2cccd40, 0xc000e2b000})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:1015 +0x585
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000599488, {0x3232400?, 0xc001e3a750?}, 0xc0012800c0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/grpc_provider.go:613 +0x497
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc0002c8500, {0x3232400?, 0xc001e3a210?}, 0xc00108b560)
	github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/tf5server/server.go:747 +0x41e
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x2c5f7e0?, 0xc0002c8500}, {0x3232400, 0xc001e3a210}, 0xc0004e4cb0, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000001e0, {0x32377e8, 0xc000ad44e0}, 0xc0007406c0, 0xc000d5af30, 0x4387e30, 0x0)
	google.golang.org/grpc@v1.50.1/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x32377e8, 0xc000ad44e0}, 0xc0007406c0, 0x0)
	google.golang.org/grpc@v1.50.1/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.50.1/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.50.1/server.go:963 +0x28a

Error: The terraform-provider-google_v4.42.0_x5 plugin crashed!

@joe-a-t
Copy link

joe-a-t commented Oct 31, 2022

So far, I have seen the issue with the following resources:

  • google_storage_bucket
  • google_storage_bucket_iam_member
  • google_organization_iam_member
  • google_service_account_iam_member
  • google_service_account
  • google_project_service
  • google_compute_subnetwork
  • google_compute_global_address
  • google_compute_firewall
  • google_compute_router

There doesn't appear to be any consistency with it though, we have a bunch of very similar root modules that are working fine.

@shuyama1
Copy link
Collaborator

shuyama1 commented Oct 31, 2022

Thanks for the feedback! Can anyone please share a Terraform configuration file that run into this issue or any steps I should follow to reproduce this error?

@joe-a-t
Copy link

joe-a-t commented Oct 31, 2022

Found a way to reproduce it with the following config:

locals {
  lifecycle_rules = [
    {
      action = "Delete"
    }
  ]
}

resource "google_storage_bucket" "bucket" {
  name                        = "test-bucket-creation-foobar"
  location                    = "US"
  uniform_bucket_level_access = true

  dynamic "lifecycle_rule" {
    for_each = local.lifecycle_rules
    content {
      action {
        type = lifecycle_rule.value.action
      }
      condition {
        days_since_noncurrent_time = 7
      }
    }
  }
}

terraform {
  required_providers {
    google = {
      version = "4.37.0"
    }
  }
}

Steps:

  1. Init and apply to create the bucket with version 4.37.0 (or lower)
  2. change the google provider version to 4.42.0 and re-init
  3. run terraform apply and receive
google_storage_bucket.bucket: Refreshing state... [id=test-bucket-creation-foobar]
╷
│ Error: Plugin did not respond
│ 
│   with google_storage_bucket.bucket,
│   on main.tf line 13, in resource "google_storage_bucket" "bucket":
│   13: resource "google_storage_bucket" "bucket" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-google_v4.42.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30daa25]

goroutine 91 [running]:
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycleRuleCondition(0xc000a9d420)
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1083 +0x45
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycle(0xc0005474a0)
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1067 +0x276
github.com/hashicorp/terraform-provider-google/google.setStorageBucket(0x3958065?, 0x43?, 0xc00176e000, {0xc0012c59e0, 0x1b}, {0xc000eb0150, 0x65})
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1493 +0xf51
github.com/hashicorp/terraform-provider-google/google.resourceStorageBucketRead(0x6093528?, {0x38cbba0?, 0xc000c03000})
	github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:732 +0x3a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x3e30660?, {0x3e30660?, 0xc0008b55c0?}, 0xd?, {0x38cbba0?, 0xc000c03000?})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:712 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000b3b180, {0x3e30660, 0xc0008b55c0}, 0xc000665450, {0x38cbba0, 0xc000c03000})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:1015 +0x585
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000111470, {0x3e30660?, 0xc0008ad5c0?}, 0xc000119f80)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/grpc_provider.go:613 +0x497
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc000564460, {0x3e30660?, 0xc0008ace40?}, 0xc0009f9680)
	github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/tf5server/server.go:747 +0x41e
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x385e520?, 0xc000564460}, {0x3e30660, 0xc0008ace40}, 0xc000ea88c0, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000001e0, {0x3e35ac8, 0xc00064e680}, 0xc000a8ca20, 0xc000b54e10, 0x4f88f90, 0x0)
	google.golang.org/grpc@v1.50.1/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x3e35ac8, 0xc00064e680}, 0xc000a8ca20, 0x0)
	google.golang.org/grpc@v1.50.1/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.50.1/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.50.1/server.go:963 +0x28a

Error: The terraform-provider-google_v4.42.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

@shuyama1
Copy link
Collaborator

shuyama1 commented Oct 31, 2022

Thanks @joe-a-t! I'm able to reproduce the error. I was trying creating the bucket with version 4.41.0 and then upgrading to 4.42.0, and it seems fine. Now I can confirm that creating with provider version 4.37.0 and upgrading to 4.42.0 causes the provider to crash. Thanks for providing the config! Working on a fix now!
Please stick to the previous version before we have the fix out.

@joe-a-t
Copy link

joe-a-t commented Oct 31, 2022

yeah, real fun one to reproduce, I had to go down to 3.90.0 then hop my way up to get the magic version number

@spideyz0r
Copy link

Similar issue here. Pinning the provider to 4.41.0 makes it work.

Error during terraform plan:

│ Error: Request cancelled
│ 
│ [redacted]
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.


Stack trace from the terraform-provider-google_v4.42.0_x5 plugin:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x24dcb65]

goroutine 1581 [running]:
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycleRuleCondition(0xc000b84700)
        github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1083 +0x45
github.com/hashicorp/terraform-provider-google/google.flattenBucketLifecycle(0xc001852640)
        github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1067 +0x276
github.com/hashicorp/terraform-provider-google/google.setStorageBucket(0x2d59a8b?, 0x44?, 0xc00023f6c0, {0xc001202ae0, 0x1c}, {0xc0003c2230, 0x65})
        github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:1493 +0xf51
github.com/hashicorp/terraform-provider-google/google.resourceStorageBucketRead(0x7f0869519140?, {0x2cccd40?, 0xc00055d000})
        github.com/hashicorp/terraform-provider-google/google/resource_storage_bucket.go:732 +0x3a7
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x3232400?, {0x3232400?, 0xc001463830?}, 0xd?, {0x2cccd40?, 0xc00055d000?})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:712 +0x178
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc000c0bb20, {0x3232400, 0xc001463830}, 0xc001179e10, {0x2cccd40, 0xc00055d000})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/resource.go:1015 +0x585
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc00000c378, {0x3232400?, 0xc001463710?}, 0xc000f14ec0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.18.0/helper/schema/grpc_provider.go:613 +0x497
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc00044a320, {0x3232400?, 0xc001463200?}, 0xc00129d440)
        github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/tf5server/server.go:747 +0x41e
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x2c5f7e0?, 0xc00044a320}, {0x3232400, 0xc001463200}, 0xc0008621c0, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.10.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:349 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0000001e0, {0x32377e8, 0xc000c30000}, 0xc001498000, 0xc000c480c0, 0x4387e30, 0x0)
        google.golang.org/grpc@v1.50.1/server.go:1340 +0xd13
google.golang.org/grpc.(*Server).handleStream(0xc0000001e0, {0x32377e8, 0xc000c30000}, 0xc001498000, 0x0)
        google.golang.org/grpc@v1.50.1/server.go:1713 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.50.1/server.go:965 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.50.1/server.go:963 +0x28a

Error: The terraform-provider-google_v4.42.0_x5 plugin crashed!

@joe-a-t
Copy link

joe-a-t commented Nov 1, 2022

Can the fix get merged and a new release cut today?

@philip-harvey
Copy link

Will there be a release with this fix in it?

@shuyama1
Copy link
Collaborator

shuyama1 commented Nov 1, 2022

The fix just got merged and will be targeting the next release 4.43.0 (potentially next Monday). Pinning the provider to version 4.41.0 should prevent from crashing for now.

@joe-a-t
Copy link

joe-a-t commented Nov 1, 2022

Any way we could do a 4.42.1 or something in the next day since this is a crash?

@paulspiegel
Copy link

Thank you for the quick fix @shuyama1! I'll second what joe commented. This is causing widespread crashes for us so a quicker release would be much appreciated.

@yamin-oanda
Copy link

@joe-a-t @paulspiegel
I believe shuyama mentioned above that you can immediately fix the crashes for now by pinning to 4.41.0

@shuyama1
Copy link
Collaborator

shuyama1 commented Nov 2, 2022

Preparing 4.42.1 release and running some tests now. If no errors occur in the tests, we'll hopefully get it released by tomorrow.

@shuyama1
Copy link
Collaborator

shuyama1 commented Nov 3, 2022

4.42.1 is out! Please let me know if the error still occurs.

@krishnakanth810
Copy link

I had same issue with Cloud Run, Cloud Functions and Spanner database.

can any one help me on this?

@shuyama1
Copy link
Collaborator

shuyama1 commented Nov 8, 2022

@krishnakanth810 Would you mind filing a new issue with the Terraform configs that run into the error as well as the debug logs? Thank you!

@philip-harvey
Copy link

@krishnakanth810 have you upgraded to provider 4.42.1?

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.