Log Router Sink writerIdentity not found when setting up pubsub.Publisher role for it. (PART 2) #10114

rrmestl-cx · 2021-09-20T22:16:05Z

Hello, apologies, I was OOO the whole week the previous issue was closed.

Makes sense that I might need to create it separately then. but an additional issue shown in the terraform output in the description above shows the new service account not being found...

module.global.module.base.google_service_account_iam_member.robm_cloud_func_test_pubsub_role: Creating...

Error: Error retrieving IAM policy for service account 'projects/ivr-divasp-sysdev-dev-01/serviceAccounts/p197201731551-682071@gcp-sa-logging.iam.gserviceaccount.com': googleapi: Error 404: Service account projects/ivr-divasp-sysdev-dev-01/serviceAccounts/p197201731551-682071@gcp-sa-logging.iam.gserviceaccount.com does not exist., notFound

  on ../global/modules/base/test_resource.tf line 82, in resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role":
  82: resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role" {

if upon execution of the terraform script, and it knows the new service account, why does it fail trying to set the role for it?

is there an execution ordering issue going on ?

Thanks,
Rob

Originally posted by @rrmestl-cx in #9953 (comment)

The text was updated successfully, but these errors were encountered:

rrmestl-cx · 2021-09-20T22:21:01Z

FYI @edwardmedia , I opened a followup issue to the previous one.

edwardmedia · 2021-09-21T00:20:43Z

@rrmestl-cx do you have the full debug log that includes the related api calls? And the new config including the new related resources

rrmestl-cx · 2021-09-21T20:14:26Z

@edwardmedia , by "full debug log" you mean the stdout of the terraform execution, or another log file?

edwardmedia · 2021-09-21T21:24:36Z

@rrmestl-cx sorry I did not make clear. Yes, I need the log generated from terraform execution. Your above one is too short and does not tell us much

rrmestl-cx · 2021-09-21T22:13:09Z

will try to run it again, and capture it all, when I get a free minute this evening.
thx!

rrmestl-cx · 2021-09-21T22:56:06Z

@edwardmedia , here is the full output for the 3 terraform calls...

$/home/isdrrm/gcp/mosaic/robm-cloud-func-test/sysdev: terraform init -upgrade

Upgrading modules...
- global in ../global
- global.base in ../global/modules/base
- us-central1 in us-central1
- us-central1.regional in ../regional
- us-central1.regional.base in ../regional/modules/base

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/google...
- Installing hashicorp/google v3.85.0...
- Installed hashicorp/google v3.85.0 (self-signed, key ID 34365D9472D7468F)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has made some changes to the provider dependency selections recorded
in the .terraform.lock.hcl file. Review those changes and commit them to your
version control system if they represent changes you intended to make.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

$/home/isdrrm/gcp/mosaic/robm-cloud-func-test/sysdev: terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.global.module.base.google_cloudbuild_trigger.robm_cloud_func_code_build_trigger will be created
  + resource "google_cloudbuild_trigger" "robm_cloud_func_code_build_trigger" {
      + create_time = (known after apply)
      + description = "testing cloud function sandbox"
      + filename    = "cloudbuild.yaml"
      + id          = (known after apply)
      + name        = "robm-cloud-func-test-trigger"
      + project     = "ivr-divasp-sysdev-dev-01"
      + trigger_id  = (known after apply)

      + github {
          + name  = "robm-cloud-func-test"
          + owner = "rrmestl-cx"

          + pull_request {
              + branch = "main$"
            }
        }
    }

  # module.global.module.base.google_logging_project_sink.robm_cloud_func_test_sink will be created
  + resource "google_logging_project_sink" "robm_cloud_func_test_sink" {
      + description            = "sink to handle RobM Cloud Func tests"
      + destination            = "pubsub.googleapis.com/projects/ivr-divasp-sysdev-dev-01/topics/robm-cloud-func-test-topic"
      + filter                 = "resource.type = gce_instance AND (textPayload =~ \"robm-cloud-func-test:\" OR textPayload =~ \"robm-cloud-func-test\\[.*?\\]: ERROR\" OR jsonPayload.message =~ \"robm-cloud-func-test:\" OR jsonPayload.message =~ \"robm-cloud-func-test\\[.*?\\]: ERROR\" OR jsonPayload.syslog_tag = \"robm-cloud-func-test\")"
      + id                     = (known after apply)
      + name                   = "robm-cloud-func-test-sink"
      + project                = (known after apply)
      + unique_writer_identity = true
      + writer_identity        = (known after apply)

      + bigquery_options {
          + use_partitioned_tables = (known after apply)
        }
    }

  # module.global.module.base.google_pubsub_subscription.robm_cloud_func_test_subs will be created
  + resource "google_pubsub_subscription" "robm_cloud_func_test_subs" {
      + ack_deadline_seconds       = 10
      + id                         = (known after apply)
      + labels                     = {
          + "state" = "active"
        }
      + message_retention_duration = "604800s"
      + name                       = "robm-cloud-func-test-sub"
      + path                       = (known after apply)
      + project                    = (known after apply)
      + retain_acked_messages      = false
      + topic                      = "robm-cloud-func-test-topic"

      + expiration_policy {
          + ttl = (known after apply)
        }
    }

  # module.global.module.base.google_pubsub_topic.robm_cloud_func_test_topic will be created
  + resource "google_pubsub_topic" "robm_cloud_func_test_topic" {
      + id      = (known after apply)
      + labels  = {
          + "state" = "active"
        }
      + name    = "robm-cloud-func-test-topic"
      + project = (known after apply)

      + message_storage_policy {
          + allowed_persistence_regions = (known after apply)
        }

      + schema_settings {
          + encoding = (known after apply)
          + schema   = (known after apply)
        }
    }

  # module.global.module.base.google_service_account_iam_member.robm_cloud_func_test_pubsub_role will be created
  + resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role" {
      + etag               = (known after apply)
      + id                 = (known after apply)
      + member             = "group:iva-alerting-sysdev@mosaic-cx.com"
      + role               = "roles/pubsub.publisher"
      + service_account_id = (known after apply)
    }

Plan: 5 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

$/home/isdrrm/gcp/mosaic/robm-cloud-func-test/sysdev: terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.global.module.base.google_cloudbuild_trigger.robm_cloud_func_code_build_trigger will be created
  + resource "google_cloudbuild_trigger" "robm_cloud_func_code_build_trigger" {
      + create_time = (known after apply)
      + description = "testing cloud function sandbox"
      + filename    = "cloudbuild.yaml"
      + id          = (known after apply)
      + name        = "robm-cloud-func-test-trigger"
      + project     = "ivr-divasp-sysdev-dev-01"
      + trigger_id  = (known after apply)

      + github {
          + name  = "robm-cloud-func-test"
          + owner = "rrmestl-cx"

          + pull_request {
              + branch = "main$"
            }
        }
    }

  # module.global.module.base.google_logging_project_sink.robm_cloud_func_test_sink will be created
  + resource "google_logging_project_sink" "robm_cloud_func_test_sink" {
      + description            = "sink to handle RobM Cloud Func tests"
      + destination            = "pubsub.googleapis.com/projects/ivr-divasp-sysdev-dev-01/topics/robm-cloud-func-test-topic"
      + filter                 = "resource.type = gce_instance AND (textPayload =~ \"robm-cloud-func-test:\" OR textPayload =~ \"robm-cloud-func-test\\[.*?\\]: ERROR\" OR jsonPayload.message =~ \"robm-cloud-func-test:\" OR jsonPayload.message =~ \"robm-cloud-func-test\\[.*?\\]: ERROR\" OR jsonPayload.syslog_tag = \"robm-cloud-func-test\")"
      + id                     = (known after apply)
      + name                   = "robm-cloud-func-test-sink"
      + project                = (known after apply)
      + unique_writer_identity = true
      + writer_identity        = (known after apply)

      + bigquery_options {
          + use_partitioned_tables = (known after apply)
        }
    }

  # module.global.module.base.google_pubsub_subscription.robm_cloud_func_test_subs will be created
  + resource "google_pubsub_subscription" "robm_cloud_func_test_subs" {
      + ack_deadline_seconds       = 10
      + id                         = (known after apply)
      + labels                     = {
          + "state" = "active"
        }
      + message_retention_duration = "604800s"
      + name                       = "robm-cloud-func-test-sub"
      + path                       = (known after apply)
      + project                    = (known after apply)
      + retain_acked_messages      = false
      + topic                      = "robm-cloud-func-test-topic"

      + expiration_policy {
          + ttl = (known after apply)
        }
    }

  # module.global.module.base.google_pubsub_topic.robm_cloud_func_test_topic will be created
  + resource "google_pubsub_topic" "robm_cloud_func_test_topic" {
      + id      = (known after apply)
      + labels  = {
          + "state" = "active"
        }
      + name    = "robm-cloud-func-test-topic"
      + project = (known after apply)

      + message_storage_policy {
          + allowed_persistence_regions = (known after apply)
        }

      + schema_settings {
          + encoding = (known after apply)
          + schema   = (known after apply)
        }
    }

  # module.global.module.base.google_service_account_iam_member.robm_cloud_func_test_pubsub_role will be created
  + resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role" {
      + etag               = (known after apply)
      + id                 = (known after apply)
      + member             = "group:iva-alerting-sysdev@mosaic-cx.com"
      + role               = "roles/pubsub.publisher"
      + service_account_id = (known after apply)
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.global.module.base.google_pubsub_topic.robm_cloud_func_test_topic: Creating...
module.global.module.base.google_cloudbuild_trigger.robm_cloud_func_code_build_trigger: Creating...
module.global.module.base.google_cloudbuild_trigger.robm_cloud_func_code_build_trigger: Creation complete after 0s [id=projects/ivr-divasp-sysdev-dev-01/triggers/17ceb106-626a-4d68-8736-6a7964334bef]
module.global.module.base.google_pubsub_topic.robm_cloud_func_test_topic: Creation complete after 1s [id=projects/ivr-divasp-sysdev-dev-01/topics/robm-cloud-func-test-topic]
module.global.module.base.google_logging_project_sink.robm_cloud_func_test_sink: Creating...
module.global.module.base.google_pubsub_subscription.robm_cloud_func_test_subs: Creating...
module.global.module.base.google_logging_project_sink.robm_cloud_func_test_sink: Creation complete after 1s [id=projects/ivr-divasp-sysdev-dev-01/sinks/robm-cloud-func-test-sink]
module.global.module.base.google_service_account_iam_member.robm_cloud_func_test_pubsub_role: Creating...
module.global.module.base.google_pubsub_subscription.robm_cloud_func_test_subs: Creation complete after 2s [id=projects/ivr-divasp-sysdev-dev-01/subscriptions/robm-cloud-func-test-sub]

Error: Error retrieving IAM policy for service account 'projects/ivr-divasp-sysdev-dev-01/serviceAccounts/p197201731551-214505@gcp-sa-logging.iam.gserviceaccount.com': googleapi: Error 404: Service account projects/ivr-divasp-sysdev-dev-01/serviceAccounts/p197201731551-214505@gcp-sa-logging.iam.gserviceaccount.com does not exist., notFound

  on ../global/modules/base/test_resource.tf line 82, in resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role":
  82: resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role" {

Let me know if you need me to lookup anything in the GCP console and provide feedback evidence for you.

Thanks,
Rob

edwardmedia · 2021-09-23T21:26:09Z

@rrmestl-cx could you add below environment variable and apply the config again? Here is the doc that explains about Terraform debugging.... Sorry I did not make it clear earlier. With that, we should be able to catch some api requests & responses

export TF_LOG=DEBUG

rrmestl-cx · 2021-09-27T19:10:14Z

@edwardmedia , not a problem, will get it ran again soon, and feedback posted.
Thanks,
Rob

rrmestl-cx · 2021-09-27T19:42:31Z

@edwardmedia , I have reran the terraform config again, with the env var in place...
attached are the plan and apply output log files...

test_20210927_plan.txt
test_20210927_apply.txt

Thanks,
Rob

edwardmedia · 2021-09-27T20:48:38Z

@rrmestl-cx from two logs you provided, I only see below two statements. Where to find the error you described here?

Plan: 4 to add, 1 to change, 0 to destroy.

Apply complete! Resources: 4 added, 1 changed, 0 destroyed.

rrmestl-cx · 2021-09-28T16:09:55Z

@edwardmedia , apologies, I had some other config code setup... let me revert and rerun the test...
Thanks,
Rob

rrmestl-cx · 2021-09-28T19:17:00Z

@edwardmedia , here are the debug output files for plan & apply for the correct config setup now...

test_20210928_plan.txt
test_20210928_apply.txt

env var set for it...

$/home/isdrrm/gcp/mosaic/robm-cloud-func-test/sysdev: echo ${TF_LOG}
DEBUG

Thanks,
Rob

edwardmedia · 2021-09-28T23:21:03Z

@rrmestl-cx where did you get the serviceAccount for writer_identity in below config? Can you verify if it exists? I can't find it in your entire context

p197201731551-137007@gcp-sa-logging.iam.gserviceaccount.com

  # module.global.module.base.google_logging_project_sink.robm_cloud_func_test_sink will be created
  + resource "google_logging_project_sink" "robm_cloud_func_test_sink" {
      + description            = "sink to handle RobM Cloud Func tests"
      + destination            = "pubsub.googleapis.com/projects/ivr-divasp-sysdev-dev-01/topics/robm-cloud-func-test-topic"
      + filter                 = "resource.type = gce_instance AND (textPayload =~ \"robm-cloud-func-test:\" OR textPayload =~ \"robm-cloud-func-test\\[.*?\\]: ERROR\" OR jsonPayload.message =~ \"robm-cloud-func-test:\" OR jsonPayload.message =~ \"robm-cloud-func-test\\[.*?\\]: ERROR\" OR jsonPayload.syslog_tag = \"robm-cloud-func-test\")"
      + id                     = (known after apply)
      + name                   = "robm-cloud-func-test-sink"
      + project                = (known after apply)
      + unique_writer_identity = true
      + writer_identity        = (known after apply)

      + bigquery_options {
          + use_partitioned_tables = (known after apply)
        }
    }

rrmestl-cx · 2021-09-29T14:57:06Z

@edwardmedia , it was taken from the resulting log router sink creation, using locals variable manipulation, and then placed into the iam_member resource to attempt setting the role for pubsub.publisher ...

locals {
   robm_cloud_func_test_sink_writer_identity_split = split(":",google_logging_project_sink.robm_cloud_func_test_sink.writer_identity)
   robm_cloud_func_test_sink_writer_identity = element(local.robm_cloud_func_test_sink_writer_identity_split, 1)
}

resource "google_service_account_iam_member" "robm_cloud_func_test_pubsub_role" {
  service_account_id = "projects/${var.project}/serviceAccounts/${local.robm_cloud_func_test_sink_writer_identity}"
  role               = "roles/pubsub.publisher"
  member             = "group:iva-alerting-${var.environment}@mosaic-cx.com"
}

This terraform excerpt was part of the original TF file attached in the previous/parent issue #9953 ...

Thanks,
Rob

edwardmedia · 2021-10-01T15:49:14Z

@rrmestl-cx yes. Below is the issue which complains the account is not found. Whatever you do, you need to make sure that account exists at that point. To debug, you can start to verify if is exists. Does this make sense?

error: Error retrieving IAM policy for service account 'projects/ivr-divasp-sysdev-dev-01/serviceAccounts/p197201731551-137007@gcp-sa-logging.iam.gserviceaccount.com': googleapi: Error 404: Service account projects/ivr-divasp-sysdev-dev-01/serviceAccounts/p197201731551-137007@gcp-sa-logging.iam.gserviceaccount.com does not exist., notFound

rrmestl-cx · 2021-10-01T22:24:51Z

@edwardmedia , yes, the writerIdentity service account does exist, as I see it, after the apply is completed, though failing...

luckily, I was able to get some support assistance through GCP support, and they found that if I used the following resource setup, it would work as expected, and complete successfully.

# Because our sink uses a unique_writer, we must grant that writer access to the bucket.
resource "google_project_iam_binding" "robm_cloud_func_test_pubsub_role" {
   role = "roles/pubsub.publisher"

   members = [
      google_logging_project_sink.robm_cloud_func_test_sink.writer_identity
   ]
}

If you have any other insight as to the resource usage I was using, that would be great... guess it was no seeing what I wanted it to...

Otherwise, I will be out of office the next week, so we can probly go ahead close this issue.

Thanks for all your assistance,
Rob

edwardmedia · 2021-10-04T15:38:43Z

@rrmestl-cx google_service_account_iam_member and google_project_iam_binding work on different resources. One is assigning IAM on a service account while the other is on the project. I am glad you have fixed the problem. Closing the issue then

github-actions · 2021-11-04T02:08:22Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

edwardmedia mentioned this issue Sep 21, 2021

Log Router Sink writerIdentity not found when setting up pubsub.Publisher role for it. #9953

Closed

edwardmedia self-assigned this Sep 21, 2021

edwardmedia added the waiting-response label Sep 21, 2021

github-actions bot removed the waiting-response label Sep 21, 2021

edwardmedia added bug waiting-response labels Sep 21, 2021

github-actions bot removed the waiting-response label Sep 21, 2021

edwardmedia added the waiting-response label Sep 23, 2021

github-actions bot removed the waiting-response label Sep 27, 2021

edwardmedia added the waiting-response label Sep 27, 2021

github-actions bot removed the waiting-response label Sep 28, 2021

edwardmedia added the waiting-response label Sep 28, 2021

github-actions bot removed the waiting-response label Sep 29, 2021

edwardmedia added the waiting-response label Oct 1, 2021

github-actions bot removed the waiting-response label Oct 1, 2021

edwardmedia added the waiting-response label Oct 1, 2021

github-actions bot removed the waiting-response label Oct 1, 2021

edwardmedia closed this as completed Oct 4, 2021

github-actions bot locked as resolved and limited conversation to collaborators Nov 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log Router Sink writerIdentity not found when setting up pubsub.Publisher role for it. (PART 2) #10114

Log Router Sink writerIdentity not found when setting up pubsub.Publisher role for it. (PART 2) #10114

rrmestl-cx commented Sep 20, 2021

rrmestl-cx commented Sep 20, 2021

edwardmedia commented Sep 21, 2021 •

edited

Loading

rrmestl-cx commented Sep 21, 2021 •

edited

Loading

edwardmedia commented Sep 21, 2021

rrmestl-cx commented Sep 21, 2021

rrmestl-cx commented Sep 21, 2021

edwardmedia commented Sep 23, 2021 •

edited

Loading

rrmestl-cx commented Sep 27, 2021

rrmestl-cx commented Sep 27, 2021

edwardmedia commented Sep 27, 2021

rrmestl-cx commented Sep 28, 2021

rrmestl-cx commented Sep 28, 2021

edwardmedia commented Sep 28, 2021 •

edited

Loading

rrmestl-cx commented Sep 29, 2021 •

edited

Loading

edwardmedia commented Oct 1, 2021

rrmestl-cx commented Oct 1, 2021

edwardmedia commented Oct 4, 2021

github-actions bot commented Nov 4, 2021

Log Router Sink writerIdentity not found when setting up pubsub.Publisher role for it. (PART 2) #10114

Log Router Sink writerIdentity not found when setting up pubsub.Publisher role for it. (PART 2) #10114

Comments

rrmestl-cx commented Sep 20, 2021

rrmestl-cx commented Sep 20, 2021

edwardmedia commented Sep 21, 2021 • edited Loading

rrmestl-cx commented Sep 21, 2021 • edited Loading

edwardmedia commented Sep 21, 2021

rrmestl-cx commented Sep 21, 2021

rrmestl-cx commented Sep 21, 2021

edwardmedia commented Sep 23, 2021 • edited Loading

rrmestl-cx commented Sep 27, 2021

rrmestl-cx commented Sep 27, 2021

edwardmedia commented Sep 27, 2021

rrmestl-cx commented Sep 28, 2021

rrmestl-cx commented Sep 28, 2021

edwardmedia commented Sep 28, 2021 • edited Loading

rrmestl-cx commented Sep 29, 2021 • edited Loading

edwardmedia commented Oct 1, 2021

rrmestl-cx commented Oct 1, 2021

edwardmedia commented Oct 4, 2021

github-actions bot commented Nov 4, 2021

edwardmedia commented Sep 21, 2021 •

edited

Loading

rrmestl-cx commented Sep 21, 2021 •

edited

Loading

edwardmedia commented Sep 23, 2021 •

edited

Loading

edwardmedia commented Sep 28, 2021 •

edited

Loading

rrmestl-cx commented Sep 29, 2021 •

edited

Loading