Provider Crash on parsing creation_time

[awatson72]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.2.9

Affected Resource(s)

N/A (All resources)

Terraform Configuration Files

N/A (Problem not related to configuration files)

N/A

Debug Output

 Error: Request cancelled
│ 
│   with module.logging_export.google_project_iam_member.llbean_project_viewer_for_logging,
│   on ../modules/project-logging/iam.tf line 60, in resource "google_project_iam_member" "llbean_project_viewer_for_logging":
│   60: resource "google_project_iam_member" "llbean_project_viewer_for_logging" {
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.logging_export.google_pubsub_topic.log_publish,
│   on ../modules/project-logging/logging.tf line 1, in resource "google_pubsub_topic" "log_publish":
│    1: resource "google_pubsub_topic" "log_publish" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-google_v3.90.1_x5 plugin:

panic: Error reading level state: strconv.ParseInt: parsing "1628102138548": value out of range

goroutine 349 [running]:
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).get(0xd210410, 0xcfcca58, 0x1, 0x1, 0xcad4308, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource_data.go:553 +0x2ba
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).State(0xd210410, 0xd210410)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource_data.go:351 +0x288
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ShimInstanceStateFromValue(0xcbf5ea0, 0xa806acc, 0xcdeda00, 0x9ef8c40, 0xd31d1a0, 0xa806acc, 0xcdeda00, 0x9ef8c40)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/resource.go:260 +0x191
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc80cee0, 0xa80650c, 0xd31c020, 0xd31c060, 0xd31c020, 0x80517f6, 0xa23f740)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-sdk/v2@v2.5.0/helper/schema/grpc_provider.go:550 +0x1ea
github.com/hashicorp/terraform-plugin-go/tfprotov5/server.(*server).ReadResource(0xcdb2cc0, 0xa80656c, 0xd31c020, 0xd463e60, 0xcdb2cc0, 0xcd9a501, 0xd20ec30)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/server/server.go:298 +0xd4
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler(0xa2cde80, 0xcdb2cc0, 0xa80656c, 0xd20ec30, 0xd463e30, 0x0, 0xa80656c, 0xd20ec30, 0xcd9a500, 0x4c0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/github.com/hashicorp/terraform-plugin-go@v0.2.1/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:344 +0x199
google.golang.org/grpc.(*Server).processUnaryRPC(0xc94c360, 0xa817028, 0xc979500, 0xcacdcc0, 0xcdb2cd8, 0xb564668, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:1297 +0x4ea
google.golang.org/grpc.(*Server).handleStream(0xc94c360, 0xa817028, 0xc979500, 0xcacdcc0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:1626 +0xa71
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xccb3910, 0xc94c360, 0xa817028, 0xc979500, 0xcacdcc0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:941 +0x92
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/pkg/mod/google.golang.org/grpc@v1.40.0/server.go:939 +0x1b0

Error: The terraform-provider-google_v3.90.1_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Panic Output

N/A

Expected Behavior

Normal Terraform plan displayed

Actual Behavior

The above error

Steps to Reproduce

1. terraform plan

Important Factoids

Other Terraform directories / states on this laptop are fine - no issue.

References

• #0000

[awatson72]

Here's an excerpt from the state file which shows that the integer reported in the output above as 'value out of range' is the creation_time:

                "user_by_email": "",
                "view": []
              }
            ],
            "creation_time": 1628102138548,
            "dataset_id": "usage_metering",
            "default_encryption_configuration": [],

Happy to provide more info if needed.

[edwardmedia]

@awatson72 what resource is it? google_pubsub_topic? But it does not have dataset_id. Can you share the config, the state and debug log?

[awatson72]

@edwardmedia I don't believe this is related to one particular resource, I don't see an option to generate a debug log while running 'terraform plan'. No debug log, or crash log is generated, only the output to the screen which I pasted above.
Here's the tfconfig:

################################################################################
## Terraform configuration, including:
##   * Backend definition
##   * Version restrictions
################################################################################

terraform {
  backend "gcs" {
    
    ## Google Cloud Storage bucket name where remote state will be stored
    bucket = "llb-terraform-preprod"

    ## (Optional) Folder name to use within the storage bucket
    # prefix      = "<storage-bucket-folder-name>"
    impersonate_service_account = "platform-terraform@llb-gcp-terraform.iam.gserviceaccount.com"

  }

  ## Restrict provider version to be above or equal to specified version, but within the major version release.
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 3.7"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 3.7"
    }
  }

}

And the provider config:

provider "google" {
  alias = "impersonation"
  scopes = [
   "https://www.googleapis.com/auth/cloud-platform",
   "https://www.googleapis.com/auth/userinfo.email",
  ]
}

data "google_service_account_access_token" "default" {
  provider               	= google.impersonation
  target_service_account 	= "platform-terraform@llb-gcp-terraform.iam.gserviceaccount.com"
  scopes                 	= ["userinfo-email", "cloud-platform"]
  lifetime               	= "1200s"
}

provider "google" {
  project     = var.gcp_project_id
  access_token	= data.google_service_account_access_token.default.access_token
  request_timeout 	= "60s"
}

provider "google-beta" {
  project     = var.gcp_project_id
  access_token	= data.google_service_account_access_token.default.access_token
  request_timeout 	= "60s"
}

If the state file is needed, I would need to share that privately or do a bit of 'sanitization'

[awatson72]

Note that I have tried a bunch of different provider versions, including the newer 4.x versions - all have the same issue.

[edwardmedia]

@awatson72 it appears on the field creation_time but I have no clue which resource it belongs to. The value of that time seem wrong. We need to understand how and when it was created. After it was already entered, all subsequent read could fail. I guess that is what are experiencing no matter what version you choose. I am looking for the logic where it is created and entered. Does this make sense?

[awatson72]

It might be relevant to report the Windows version that I'm using WSL on Windows 10 21H2.

[awatson72]

@awatson72 it appears on the field creation_time but I have no clue which resource it belongs to. The value of that time seem wrong. We need to understand how and when it was created. After it was already entered, all subsequent read could fail. I guess that is what are experiencing no matter what version you choose. I am looking for the logic where it is created and entered. Make sense?

Yep - I follow you. I assumed 'creation_time' applied to the whole state file, but let me dig deeper....

[edwardmedia]

@awatson72 it is very likely. From the log you showed above, it has nothing to do with the provider

[awatson72]

ok, so it appears to be related to this resource:

################################################################################
## BQ
################################################################################
resource "google_bigquery_dataset" "google-gke-usage-metering-dataset" {
  dataset_id    = "usage_metering"
  friendly_name = "Google GKE Usage Metering"
  description   = "Google GKE Usage Metering"
  location      = "US"
}

The state file:

  {
      "mode": "managed",
      "type": "google_bigquery_dataset",
      "name": "google-gke-usage-metering-dataset",
      "provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "access": [
              {
                "domain": "",
                "group_by_email": "",
                "role": "OWNER",
                "special_group": "",
                "user_by_email": "platform-terraform@llb-gcp-terraform.iam.gserviceaccount.com",
                "view": []
              },
              {
                "domain": "",
                "group_by_email": "",
                "role": "OWNER",
                "special_group": "projectOwners",
                "user_by_email": "",
                "view": []
              },
              {
                "domain": "",
                "group_by_email": "",
                "role": "READER",
                "special_group": "projectReaders",
                "user_by_email": "",
                "view": []
              },
              {
                "domain": "",
                "group_by_email": "",
                "role": "WRITER",
                "special_group": "",
                "user_by_email": "service-561578754233@gcp-sa-bigquerydatatransfer.iam.gserviceaccount.com",
                "view": []
              },
              {
                "domain": "",
                "group_by_email": "",
                "role": "WRITER",
                "special_group": "projectWriters",
                "user_by_email": "",
                "view": []
              }
            ],
            "creation_time": 1628102138548,
            "dataset_id": "usage_metering",
            "default_encryption_configuration": [],
            "default_partition_expiration_ms": 0,
            "default_table_expiration_ms": 0,
            "delete_contents_on_destroy": false,
            "description": "Google GKE Usage Metering",
            "etag": "e1rJbtNkFYWALL4JocGR9Q==",
            "friendly_name": "Google GKE Usage Metering",
            "id": "projects/llb-gcp-preprod/datasets/usage_metering",
            "labels": {},
            "last_modified_time": 1628176989818,
            "location": "US",
            "project": "llb-gcp-preprod",
            "self_link": "https://bigquery.googleapis.com/bigquery/v2/projects/llb-gcp-preprod/datasets/usage_metering",
            "timeouts": null
          },
          "sensitive_attributes": [],
          "private": "eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoyNDAwMDAwMDAwMDAsImRlbGV0ZSI6MjQwMDAwMDAwMDAwLCJ1cGRhdGUiOjI0MDAwMDAwMDAwMH19",
          "dependencies": [
            "data.google_service_account_access_token.default"
          ]
        }
      ]
    }

[edwardmedia]

@awatson72 are you able to test run on a macbook to see if that happens?

[awatson72]

The issue does not occur on a macbook.

[rileykarson]

This has come up in the past, #6518 / hashicorp/terraform-plugin-sdk#469

[rileykarson]

(whoops, just meant to click comment)

[edwardmedia]

#6518

[awatson72]

Interesting - I'm using this terraform version:

Terraform v1.2.9
on linux_386

On this WSL

Linux L7QJT733-LP 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

For the most part, it works fine. I'll try a 64 bit version a terraform when I have chance.

[awatson72]

This issue is resolved when using the amd_64 version of terraform. So the takeaway is that Windows WSL users might be able to use the linux_386 version of terraform, but could have problems in certain circumstances.

Terraform v1.2.9
on linux_amd64

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.