Update Alertmanager Configmap when odh-parameters secret is updated

[anishasthana]

Signed-off-by: Anish Asthana anishasthana1@gmail.com

• The commits are squashed in a cohesive manner and have meaningful messages.
• For commits that came from upstream, [UPSTREAM] has been prepended to the commit message
• JIRA link(s): https://issues.redhat.com/browse/RHODS-1974
• The Jira story is acked
• An entry has been added to the latest build document in Build Announcements Folder.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious)

Test Instructions:

1. Deploy RHODS oeprator normally via the live build image.
2. Once all pods are up and running, look at the prometheus pod in redhat-ods-monitoring
3. Note the value of the alertmanager annotation
4. In the same namespace, look at the to emails next to user-notifications in the alertmanager configmap.
5. Now, update the notification-email key-value pair withing addon-managed-odh-parameters secret in the redhat-ods-operator namespace to have a new email.
6. After a few seconds ( you can look at pod logs to see when it happen), you will notice the prometheus pod restart
7. Look at the same annotation as before, it should have a different value now (you don't need to decrypt the base64 stuff)
8. The alertmanager configmap should have been updated too to be pointed at the new user-notification email

Related PR: red-hat-data-services/odh-deployer#226

[anishasthana]

Live build is available at quay.io/anishasthana/rhods-operator-live-catalog:1.102.0-1974.

[anishasthana]

Lateste live build: quay.io/anishasthana/rhods-operator-live-catalog:1.102.0-1974

[anishasthana]

/retest

[VaishnaviHire]

I think the changes that are not part of this feature implementation should be reverted

[lucferbux]

/lgtm

Although, there are a few blocks of code that I'm not sure if they're related to this feature, if there's something not related to the feature it would be nice to remove it.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anishasthana, lucferbux

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [anishasthana]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[anishasthana]

/retest

[openshift-ci]

@anishasthana: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/modh-operator-e2e	5bfa591	link	true	/test modh-operator-e2e
ci/prow/48-modh-operator-e2e-48	5bfa591	link	true	/test 48-modh-operator-e2e-48

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[VaishnaviHire]

/lgtm

Worked well when updated the emails through UI

[jgarciao]

lgtm too!