Monitoring and Alerting requirements for existing managed stack

[VaishnaviHire]

This is a tracker issue to capture requirements, changes for Monitoring stack

Requirements

• Migrate all alerting and monitoring resources from odh-deployer
• Setup alerts only when a component is in Managed state
• Scope changes required to move to new monitoring stack.
• Identify new components and additional alerts required.

Changes

• Update monitoring per component enabled/disabled red-hat-data-services/rhods-operator#67
  • split monitoring from operator git repo into small chunks for each components
  • move "monitoring + anaconda" from odh-manifests into rhods-operator
  • add support in dockerfile to include "monitoring" from operator
  • add distributed new monitoring
• change logic of checking platform: Use catalogsource to identify managed Platform #627
• apply fix from previous test [rhods-1.33]Fix wrong manifests file name, add logger, apply prometheus CM first red-hat-data-services/rhods-operator#16
• sync logic of monitoring in ODH [cherry-pick] Add logic for applying monitoring stack #618
• cleanup "monitoring + anaconda" in odh-manifests which has been duplicated in operator repo Remove monitoring folder , has been moved into rhods-operator's config folder red-hat-data-services/odh-manifests#478

[zdtsw]

ref: #341

[zdtsw]

looks like in prometheusv2.43.0https://github.com/prometheus/prometheus/releases/tag/v2.43.0 the feature has been added to support "scrape_config_files"
but the "latest" image we can use registry.redhat.io/openshift4/ose-prometheus@sha256:2572eff86dc111203c5de6ce7325cc061ddb4bd737e09a1639313f3a9ed07a9c was built off 2.42.0

[zdtsw]

after a first test with QE, some parts need to figure:

• when user update notification mail, how this will be reflact to the configmap? need to understand how v1 did this
• blackbox if this need to be split to component-based config? seems it is adding label of components?
  https://github.com/red-hat-data-services/odh-manifests/blob/34e19828049600cb72cf75852c2cfc33afe05609/monitoring/prometheus/prometheus-configs.yaml#L792C18-L792C26 is point to https://github.com/red-hat-data-services/odh-manifests/blob/34e19828049600cb72cf75852c2cfc33afe05609/monitoring/blackbox-exporter/external/blackbox-exporter-external.yaml#L9

[VaishnaviHire]

Hi @zdtsw for -

when user update notification mail, how this will be reflact to the configmap? need to understand how v1 did this

This will be a separate PR that should add the changes to watch the addn-parameters configmap. For v1 operator the change was added here - red-hat-data-services#133

[VaishnaviHire]

@zdtsw Can we close this as completed?