fix: recreate a notification channel if it doesn't exist in grafana

[weisdd]

Description

Currently, a notification channel in grafana gets recreated by the operator only in two cases:

• knownHash is empty (happens when operator is restarted);
• a hash of the notification channel is different due to changes in CR.

The PR makes operator always check whether a notification channel exist in grafana. If not, it gets submitted again.
Also, it fixes a log message delete result was %v. Previously, the code would try to substitute %v with a UID which would always be empty. Now, it uses Message from grafana response.
The logic is somewhat similar to what we have in the grafana dashboard controller and pipeline.

Relevant issues/tickets

Fixes: #811

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

• This change requires a documentation update
• I have added tests that prove my fix is effective or that my feature works
• I have added a test case that will be used to verify my changes
• Verified independently on a cluster by reviewer

Verification steps

Preparation

Like described in the issue #811, deploy grafana with unified alerting disabled:

    # Ensure LEGACY alerting
    alerting:
      enabled: true
    unified_alerting:
      enabled: false

NOTE: Unified alerting seems to be introduced in grafana 8.x, so should be no need to enable it on earlier versions. I tested on 9.0.2 since the latest images are built with M1 support.

Deploy a notification channel CR like in our deploy example:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaNotificationChannel
metadata:
  name: pager-duty-channel
  labels:
    app: grafana
spec:
  name: pager-duty-channel.json
  json: >
    {
      "uid": "pager-duty-alert-notification",
      "name": "Pager Duty alert notification",
      "type":  "pagerduty",
      "isDefault": true,
      "sendReminder": true,
      "frequency": "15m",
      "disableResolveMessage": true,
      "settings": {
        "integrationKey": "put key here",
        "autoResolve": true,
        "uploadImage": true
     }
    }

Test cases

1. Create CR
   => created

2. Update CR
   => updated

3. Delete the notification channel from grafana interface
   => recreated

4. Restart operator
   => updated with the same spec

5. Delete CR
   => deleted

[pb82]

Verified locally