ADR required for monitoring setup

[HumairAK]

We have made some progress with setting up monitoring, and have some issues already created in the appropriate repos for creating service monitors, prometheus / grafana deployments etc. What's lacking is a document that adds context to our setup and future plans. For this we should prepare an ADR that outlines our monitoring architecture.

[HumairAK]

@hemajv see if the adr doc for github alerts can be incorporated here

[4n4nd]

Since we have decided (relevant issue) to keep a single Prometheus instance that monitors all the other component namespaces.

Now we need to decide where to keep our service-monitors/pod-monitors resources.

For the ODH components these resources will probably go in the upstream odh-manifests but we will still need to add the following overlay to the upstream monitors:

spec:
    namespaceSelector:
        matchNames:
          - opf-observatorium # The namespace where the component is deployed in

[HumairAK]

If the service monitors and pod monitors are to go in the same namespace as prometheus, then I think the monitoring folder makes sense. I think we can put them in the base since they aren't environment specific, then inherit them in all the overlays -- wdyt?

[4n4nd]

@HumairAK @anishasthana I was thinking of keeping just one servicemonitor resource which looks into multiple namespaces. Maybe we can make this change upstream and only overlay the list of namespaces in operatefirst overlays?

[HumairAK]

I like the idea of having one servicemonitor for odh,

It would be nice if we could specify a list of namespaces as a parameter in the kfdef that would be populated in the servicemonitor so we don't need to do an override.

[anishasthana]

I am not opposed to it, but we should ask the monitoring team what they think. I wonder if it's abusing the idea of service monitors.

[4n4nd]

I was able to test a service monitor that was able to monitor different services in multiple namespaces.

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: multiservice-monitor
  labels:
    k8s-app: prometheus
  namespace: opf-stage
spec:
  endpoints:
    - port: metrics
    - port: 8080-tcp
  namespaceSelector:
    matchNames:
      - opf-observatorium
      - opf-jupyterhub
  selector: {}

This works well. I think we should try to get something like this pushed upstream and only set the namespaceSelector as an overlay on the operatefirst side.

[hemajv]

@hemajv see if the adr doc for github alerts can be incorporated here

I think it would be easier to have a separate ADR for this.
created #16, for adding the ADR of our alerting setup