Update monitoring per component enabled/disabled

[zdtsw]

ref: opendatahub-io#357

whats in the PR

• add ods-configs manifests into operator image
• add monitoring manifests into operator image
• update dockerfile to get above manifests from operator repo directly
• add watch on monitoring addon-management-odh-pamaters and configmap alertmanager
• move witheventfilter predicates into each owns to prevent unnecessary never-stop reconcile
• add new watch from DSCI to DSC on deletion
• when no DSC instance in cluster, prometheus should not have component rules in config
• add methods in component interface to write component rules into prometheus.yaml
• revert encode part of the values in prometheus.yaml since it need plain text
• add function MatchLineInFile to inject values in file than replace placeholder
• cherry-pick form ODH: pr 597, 627, 583

Update from 19th Oct 2023

quay.io/wenzhou/opendatahub-operator-catalog:v2.10.67-7
test case 4: no DSC, should no component rules too
create DSCI first, check prometheus config: 2 rules
create DSC with default managed components, check prometheus config: these component rules are added
delete DSC , check prometheus config: 2 rules

Update from 16th Oct 2023

quay.io/wenzhou/opendatahub-operator-catalog:v2.10.67-6

test case 3: run on a self-managed cluster
after DSCI instacne is created, should not have prometheus created.
see controllers.DSCInitialization Monitoring enabled, won't apply changes {"cluster": "Self-Managed RHODS Mode"} in the log
odh-segment-key-config CM should be created in applicationamespace.

Update from 13th Oct 2023

quay.io/wenzhou/opendatahub-operator-catalog:v2.10.67-5

test case1: email update

• install from clean cluster (should have it either as managed cluster or mock it to be a managaed cluster with csv)
• check DSCI is created, no DSC should be created.
• wait till operator pod stopped reconcile, should see log "Success: finish config managed monitoring stack!"
• update user email notification field/update secret directly
• should see operator pod start to reconcile.
• after a while go to check alertmanage CM is updated with new email
• go check Prometheus pod should be restarted

test case2: component plugin-plugout

• install DSC
• go check Prometheus config, and see only "operator" and "deadmansitch" rules are there
• enable a bunch of components (kserve wont be useful in this testcase)
• go check Prometheus config again and see new rules for the enabled component have been added
• same for the rules, only the rules for enabled components should be there: rhods metrics can be different, if you are on a mocked cluster it might say "UNKNOWN" in status
• disable some components
• go check Prometheus config again, it should reflect to the rule list
• same for the rules

Update:

newest image
quay.io/wenzhou/opendatahub-operator-catalog:v2.10.67-1 => with some fix and also enabled monitoring for self and managed both

new catsrc image is:
quay.io/wenzhou/opendatahub-operator-catalog:v2.10.67 => monitoring is enabled on the Managed cluster

for testing purpose:
quay.io/wenzhou/opendatahub-operator-catalog:v2.10.67-0 => this enabled monitoring on both Self-Managed and Managed cluster.
should see in log "Monitoring enabled, should't apply changes but for test purpose we enable it here"

Old:

old test image for myself, when mocking to use for SelfManaged cluster
local build: quay.io/wenzhou/opendatahub-operator-catalog:v2.10.2-17
debug info in the screenshots wont be available to check in the new catsrc build.

when enable workbenches, from operator pod log:

check configmap:

then disable workbenches:

check configmap (this might take a minute to get updated)

[VaishnaviHire]

Tested successfully with ocp 4.13

/lgtm