[prometheus][deckhouse] Add supporting ServiceMonitors and PodMonitors from user-space

[name212]

Description

Change podMonitorNamespaceSelector and serviceMonitorNamespaceSelector from label heritage: deckhouse to new custom prometheus.deckhouse.io/monitor-watcher-enabled label. Because we do not want give an opportunity for usage heritage: deckhouse label for non-deckhouse namespaces, but we need pass namespace selectors into Prometheus resource. We can not implement OR logic with LabelSelector and we should change labels.

Add prometheus.deckhouse.io/monitor-watcher-enabled to d8-monitoring namespace. All own Pod and Service Monitor located in d8-monitoring namespace.

Add FAQ for connect user-space Service/Pod Monitors to Prometheus.

Change alert description about custom monitors.

Custom monitors alert logic did not change, because it already has correct logic.

Why do we need it, and what problem does it solve?

There is no way to connect custom ServiceMonitor to Prometheus from Deckhouse. However, all modern helm charts are bundled with a ServiceMonitor. Users have to alter charts to provide monitoring.

Changelog entries

module: prometheus
type: feature
description: Add supporting ServiceMonitors and PodMonitors from user-space
note: Prometheus will be restarted