Terraform plan after apply has additional ACL changes required and never resolves

[jdstone]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Summary

After running terraform apply, a subsequent terraform plan indicates it needs to perform additional changes for ACLs even though the plan indicated it completed. (see output samples below)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
google_storage_bucket.example-artifacts: Refreshing state... [id=example-artifacts]
google_storage_bucket_acl.example-artifacts: Refreshing state... [id=example-artifacts-acl]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_storage_bucket_acl.example-artifacts will be updated in-place
  ~ resource "google_storage_bucket_acl" "example-artifacts" {
        id          = "example-artifacts-acl"
      ~ role_entity = [
          - "OWNER:group-engineering@example.com",
          - "READER:user-348419557177-compute@developer.gserviceaccount.com",
            "OWNER:project-owners-348419557177",
            # (1 unchanged element hidden)
            "READER:project-viewers-348419557177",
          + "OWNER:group-engineering@example.com",
            "OWNER:user-dev-ccie-gcr-upload-key@example-production.iam.gserviceaccount.com",
            "OWNER:project-editors-398082806114",
          + "OWNER:user-348419557177-compute@developer.gserviceaccount.com",
          + "READER:user-348419557177-compute@developer.gserviceaccount.com",
        ]
        # (1 unchanged attribute hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Terraform Version

Terraform v0.14.8
+ provider registry.terraform.io/hashicorp/google v3.60.0

Affected Resource(s)

• google_storage_bucket_acl

Terraform Configuration Files

provider "google" {
  credentials = file(var.google_credentials_file)
  project     = var.google_project_id
  region      = var.google_region
}

resource "google_storage_bucket" "example-artifacts" {
  name          = "example-artifacts"
  location      = "US"
  storage_class = "MULTI_REGIONAL"

  lifecycle_rule {
    action {
      type = "Delete"
    }

    condition {
      age        = "365"
      with_state = "ANY"
    }
  }
}

resource "google_storage_bucket_acl" "example-artifacts" {
  bucket = google_storage_bucket.example-artifacts.name

  role_entity = [
    "OWNER:project-owners-${var.google_project_number}",
    "OWNER:project-editors-${var.google_project_number}",
    "READER:project-viewers-${var.google_project_number}",
    "OWNER:group-engineering@example.com",
    "OWNER:user-dev-ccie-gcr-upload-key@example-production.iam.gserviceaccount.com",
    "OWNER:project-editors-398082806114",
    "OWNER:user-${var.google_project_number}-compute@developer.gserviceaccount.com",
    "READER:user-348419557177-compute@developer.gserviceaccount.com",
  ]
}

Debug Output

https://gist.github.com/jdstone/924c882f8b1f0fbf634fa9e3ffb4a663

Panic Output

N/A

Expected Behavior

It should not think it needs to make additional changes after an apply is run.

Actual Behavior

After configuring the ACLs, which seems to work, it thinks it needs to update the ACLs again.

Steps to Reproduce

1. terraform plan
2. terraform apply
3. terraform plan

Important Factoids

N/A

References

• Terraform plan after apply has additional ACL changes required and never resolves. #892
• Ignore role_entity order in storage_bucket_acl #1525 (may be related as well)

[megan07]

Hi @jdstone!

I'm sorry you're running into this. I was unable to reproduce it exactly, however, I did see similar behavior when I tried to assign OWNER and READER to the same user. It appears in the plan output that you have above that this might be the case

          + "OWNER:user-348419557177-compute@developer.gserviceaccount.com",
          + "READER:user-348419557177-compute@developer.gserviceaccount.com",

In the given configuration, I see one is a project_number variable, and the other is hard-coded, could you double-check that these are different project numbers?

    "OWNER:user-${var.google_project_number}-compute@developer.gserviceaccount.com",
    "READER:user-348419557177-compute@developer.gserviceaccount.com",

Please, let me know if they are indeed different users and you are still experiencing this.

Thanks!

[jdstone]

Hi @megan07,

Interesting, I didn't catch that. Thanks for pointing that out, but that did not fix the issue. Here is the updated config and the result of the plan after applying.

resource "google_storage_bucket_acl" "example-artifacts" {
  bucket = google_storage_bucket.example-artifacts.name

  role_entity = [
    "OWNER:project-owners-${var.google_project_number}",
    "OWNER:project-editors-${var.google_project_number}",
    "READER:project-viewers-${var.google_project_number}",
    "OWNER:group-engineering@example.com",
    "OWNER:user-dev-ccie-gcr-upload-key@example-production.iam.gserviceaccount.com",
    "OWNER:project-editors-398082806114",
    "OWNER:user-${var.google_project_number}-compute@developer.gserviceaccount.com",
  ]
}

# google_storage_bucket_acl.example-artifacts will be updated in-place
~ resource "google_storage_bucket_acl" "example-artifacts" {
      id          = "example-artifacts-acl"
    ~ role_entity = [
        - "OWNER:group-engineering@example.com",
        - "READER:user-348419557177-compute@developer.gserviceaccount.com",
          "OWNER:project-owners-348419557177",
          # (1 unchanged element hidden)
          "READER:project-viewers-348419557177",
        + "OWNER:group-engineering@example.com",
          "OWNER:user-dev-ccie-gcr-upload-key@example-production.iam.gserviceaccount.com",
          "OWNER:project-editors-398082806114",
        + "OWNER:user-348419557177-compute@developer.gserviceaccount.com",
      ]
      # (1 unchanged attribute hidden)
  }

[megan07]

Sure! And sorry again, I want to clarify that with this plan output, it is the plan after an apply of the new config? I notice it's trying to change the user's role from READER to OWNER so I want to double-check.

[jdstone]

Yes, that is correct -- this is the plan after an apply of the new config.

[megan07]

Hi @jdstone ! Would I be able to get the debug logs for your latest change?

Stepping through the code, we do something a little different with showing the changes in this resource. We compare all the role_entities in state with whats in the configuration, if their lengths are different (the initial situation) we show the entire diff, and if there is an entry in state that is not in the config (this last situation, READER:user-348419557177-compute@developer.gserviceaccount.com is in state, but not in the config) we show the whole diff. So I think if we can figure out why the backend wants to keep that user a READER rather than updating it to an OWNER, this diff will go away.

Unfortunately, I'm unable to reproduce it locally, and from the previous debug logs, the only POST I saw was to make that user a READER, so I'd like to see that there's a POST going through to make it an OWNER now. I'm curious if that's in the new debug logs?

Thanks!

[jdstone]

My apologies for the delayed response @megan07.

Here is a link to the debug logs for my latest change:
https://gist.github.com/jdstone/a270da41aa63f007e856ce62adb4d1e0

[megan07]

Hi @jdstone - I'm so sorry, I want to double-check again, is this the debug from the plan or the apply? I don't see a POST in here, so I'm assuming it's from the plan. Would I be able to see a debug log from the apply please? Sorry for all the back and forth!

[jdstone]

@megan07, this is from the terraform apply.

[megan07]

@jdstone sorry for all the back and forth, i was able to reproduce it now and have a fix coming! thank you for your patience!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!