Add retries for operations to account for intermittent network issues

[AnirudhaS]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

Currently resource for instance creation or table creation for Bigtable resources do not seem to rely on retrying requests to account for intermittent network failures.

Consider a Terraform task to setup a bigtable instance with column family configuration. This results in multiple requests, and if additional requests fail, it results in complete task failure. Moreover, to complete the task involves having to delete the tables manually, as task fails due to since the table already exists.

• https://github.com/terraform-providers/terraform-provider-google/blob/master/google/resource_bigtable_instance.go

New or Affected Resource(s)

• google_bigtable_table

Potential Terraform Configuration

resource google_bigtable_table "table" {
  name          = replace(var.name, "some-instance", "")
  project       = var.project
  instance_name = var.instance_name
  split_keys    = var.split_keys
  dynamic column_family {
    for_each = var.column_families
    content {
      family = column_family.value
    }
  }
}

References

[kevinsi4508]

This issues seems to have two parts:

1. Resource creation should be tried on transient errors.
2. Handling already_exists error.

We still have some work to do with retrying transient error, but application level retry should be sufficient.
For example KCC periodically retries/reconciles the resource.

Handling already_exists error might be something we can do.

I think it was something like this:

0. the resource was created
1. on error removed resource: clear ID.
2. TF apply again got already_existed error.

In GoogleCloudPlatform/magic-modules#6735, we will remove the resource only if there is a NOT_FOUND error. This should prevent the resources being cleared on transient errors (or any error other than NOT_FOUND).

[melinath]

b/259278591

[kevinsi4508]

We have more work to do in term of error handling. However, with GoogleCloudPlatform/magic-modules#6735, we don't clear the resource on errors other than NOT_FOUND. This should fix the reported problem. We should close this issue if no objection.

[melinath]

Closing the issue - if there are additional similar issues please open a new ticket.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.