Slack Notification Channel creation using incoming webhook no longer works

[ptran32]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.4.4
on darwin_amd64

• provider registry.terraform.io/hashicorp/google v4.60.2

Affected Resource(s)

• google_monitoring_notification_channel

Terraform Configuration Files

resource "google_monitoring_notification_channel" "default" {
  display_name = "Test Slack Channel"
  type         = "slack"
  labels = {
    "channel_name" = "#alerts-data-systems"
  }
  sensitive_labels {
    auth_token = "https://hooks.slack.com/services/XXXSECRETXXX/XXXSECRETXXX"
  }
}

Debug Output

Panic Output

https://gist.github.com/ptran32/41d75161a53d7fd68c06309d4d5b8460 (Slack sensitive outputs were hidden)

Expected Behavior

Terraform should be able to create the Google notification channel resource using a incoming webhook URL as a key

It used to work using Terraform 1.1.4, I still have a state file to share if needed. (I don't have the provider's version though)

Actual Behavior

Terraform crashes with the error: Error creating NotificationChannel: googleapi: Error 400: invalid_auth

Steps to Reproduce

1. terraform apply

Important Factoids

N/A

References

• Failing test(s): TestAccMonitoringNotificationChannel_update*Labels_slack #14081 - I wonder if this opened issue has any correlation with the problem I'm facing
• b/277768332

[edwardmedia]

@ptran32 it looks like the API did add the validation on auth_token field around 3/2/2023. It has nothing to do with the Terraform upgrade. Above test failure is expected as an incorrect value was provided.

You indicated it used to work. Does it mean the gcp resource was successfully created or the Slack channel was working? Can you double check if the value of auth_token you provided is correct? Since the Slack is the 3rd party tool, the terraform provider has no control over it. The error you are receiving is clearly pointing to that field. Can you manually config the channel to make it working?

[ptran32]

Hi @edwardmedia ,

1. Slack channel:

• was successfully created
• did not work (couldn't receive notification in the channel because of Slack Notification Channel creation via terraform no longer works #11346), but this is another topic. This is actually what I wanted to test, and see if it's still an issue

2. I double-checked, the auth_token I provided is correct

3. I manually configured the channel to make it work already (as a workaround)

[edwardmedia]

@ptran32 since you have already made it working manually, can you import google_monitoring_notification_channel to see what it has on the server?

[ptran32]

This the the output of the Terraform state file

{
    "version": 4,
    "terraform_version": "1.4.4",
    "serial": 2,
    "lineage": "66d8bbae-f023-7cfb-d2b7-a09ed828d689",
    "outputs": {},
    "resources": [
      {
        "mode": "managed",
        "type": "google_monitoring_notification_channel",
        "name": "default",
        "provider": "provider[\"registry.terraform.io/hashicorp/google\"]",
        "instances": [
          {
            "schema_version": 0,
            "attributes": {
              "description": "",
              "display_name": "alerts-data-systems",
              "enabled": true,
              "force_delete": false,
              "id": "projects/just-data-preprod/notificationChannels/1295324048836xxx",
              "labels": {
                "auth_token": "",
                "channel_name": "#alerts-data-systems",
                "team": "Tech"
              },
              "name": "projects/just-data-preprod/notificationChannels/1295324048836xxx",
              "project": "just-data-preprod",
              "sensitive_labels": [],
              "timeouts": {
                "create": null,
                "delete": null,
                "update": null
              },
              "type": "slack",
              "user_labels": {},
              "verification_status": ""
            },
            "sensitive_attributes": [],
            "private": "sensitivevalueXXX"
          }
        ]
      }
    ],
    "check_results": null
  }
`

[edwardmedia]

@ptran32 is this state from the imported resource? I see below attribute empty. Maybe you can share the debug log for import?

"sensitive_labels": [],

To be clear, my understanding is you have made the channel working. By import the working resource, we might be able to inspect what sensitive_labels is saved on the server. I don't understand the api accepts the value for auth_token via console but not from terraform.

[ptran32]

Yes, it is the state from the imported resource using:

terraform import google_monitoring_notification_channel.default projects/just-data-preprod/notificationChannels/129532404883688xxxx

Just to be sure we're aligned with the steps I've done before, here's more details:

• Create a Slack notification channel manually via Google UI (as a workaround because resource creation via Terraform doesn't work)
• Test and confirm that alerts can be received to this channel
• Import the state using the command posted above (see logs of the import in debug verbosity)