[Neutron] Alert on mariadb endpoint not ready

[sven-rosenzweig]

Trigger a warning alert if mariadb endpoint is not ready.

Restarting neutron-server during the mariadb
endpoint not being ready, results in a failing neutron-server pods. The kubernetes-entrypoint script checks if all dependencies (e.g. mariadb reachable) are fullfield before starting neutron.

To detect such situations, this alert fires if such a situation occurs again.

[sapcc-bot]

Failed to validate the Prometheus rules. Details. Readme.

[sven-rosenzweig]

following issue occured during a deployment:

• readiness check of mariadb failed --> k8s takes down mariadb endpoint
• deployment restarted neutron-server
• kubernetes-entrypoint script prevented them from starting since dependency to mariadb not given ) kubernetes-entrypoint

Neutron-Server Log

Entrypoint WARNING: 2024/04/30 13:36:58 entrypoint.go:71: Resolving dependency Service neutron-mariadb in namespace monsoon3 failed: Service neutron-mariadb has no endpoints

k describe endpoints neutron-mariadb
Name:         neutron-mariadb
Namespace:    monsoon3
Labels:       app.kubernetes.io/managed-by=Helm
              ccloud/support-group=network-api
              component=neutron
              system=openstack
              type=database
Annotations:  endpoints.kubernetes.io/last-change-trigger-time: 2024-05-02T10:58:25Z
Subsets:
  Addresses:          <none>
  NotReadyAddresses:  100.90.18.188
  Ports:
    Name             Port  Protocol
    ----             ----  --------
    neutron-mariadb  3306  TCP
    readiness        8080  TCP
Events:  <none>

Failing Readiness Check

Events:
  Type     Reason     Age   From     Message
  ----     ------     ----  ----     -------
  Warning  Unhealthy  21m   kubelet  Readiness probe failed: Get "http://100.90.18.188:8080/healthy": read tcp 100.90.18.1:59816->100.90.18.188:8080: read: connection reset by peer