google_bigtable_instance is incorrectly removed from state

[ericnorris]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.12.31
+ provider.google (3.81.0)

Affected Resource(s)

• google_bigtable_instance

Terraform Configuration Files

resource "google_bigtable_instance" "some-instance" {
  name = "some-instance"

  cluster {
    cluster_id   = "some-cluster-c1"
    zone         = "us-central1-c"
    storage_type = "SSD"
  }

}

Debug Output

2021-09-16T14:41:05.659Z [INFO]  plugin.terraform-provider-google_v3.81.0_x5: 2021/09/16 14:41:05 [WARN] Removing some-instance because it's gone: timestamp=2021-09-16T14:41:05.659Z

Panic Output

N/A

Expected Behavior

• The plan should have failed due to the underlying error.
• The request & response logs (including the error) should be visible when TF_LOG=trace is set.

Actual Behavior

• The resource was presumed "gone", and it was planned to be created again:

# google_bigtable_instance.some-instance will be created
  + resource "google_bigtable_instance" "some-instance" {
      + deletion_protection = true
      + display_name        = (known after apply)
      + id                  = (known after apply)
      + instance_type       = "PRODUCTION"
      + name                = "some-instance"
      + project             = (known after apply)

      + cluster {
          + cluster_id   = "some-cluster-c1"
          + kms_key_name = (known after apply)
          + num_nodes    = (known after apply)
          + storage_type = "SSD"
          + zone         = "us-central1-c"
        }
    }

• There were no request / response logs for the failing API call when TF_LOG=trace was set.

Steps to Reproduce

1. terraform apply
2. Do something to cause https://github.com/hashicorp/terraform-provider-google/blob/release-3.81.0/google/resource_bigtable_instance.go#L216 to fail, e.g. disable the bigtable API.
3. terraform plan

Important Factoids

I spent hours trying to figure this one out.

The underlying cause for us was that the Bigtable client didn't respect the configured user_project_override and billing_project, and was encountering this error:

rpc error: code = PermissionDenied desc = Cloud Bigtable Admin API has not been used in project <project-number> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/bigtableadmin.googleapis.com/overview?project=<project-number> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

This has been fixed by #10060, which has not yet been incorporated into a release. It was really tough to debug as the request / response logs for these API calls do not appear to be logged by the Bigtable instance client, so it'd be nice if a fix for this issue included ensuring that the Bigtable client is using the HTTP client that is configured to log requests and responses [1].

Ignoring that, I believe that:

terraform-provider-google/google/resource_bigtable_instance.go

Lines 217 to 221 in a5e5535

	if err != nil {
	log.Printf("[WARN] Removing %s because it's gone", instanceName)
	d.SetId("")
	return nil
	}

...should be adjusted to only assume the resource is gone on a 404. There is precedent in other resources, e.g:

terraform-provider-google/google/resource_google_service_account_key.go

Lines 164 to 167 in 5c3dfa2

	if isGoogleApiErrorWithCode(err, 403) {
	log.Printf("[DEBUG] Got a 403 error trying to read service account key %s, assuming it's gone.", d.Id())
	d.SetId("")
	return nil

I'd also encourage any other places where this happens to be moved to this pattern where possible.

[1]

terraform-provider-google/google/config.go

Line 78 in a5e5535

client *http.Client

References

• Backfill user_project_overrides + billing project for Bigtable resources #10060

[kevinsi4508]

I was able to reproduce this. It does seems to be a bug. I will investigate further to see what should be done. Thanks for raising this.

[melinath]

I think we've had the issue of logging come up before - possibly #12159?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.