Customize Prometheus label selector #3356

alexandre-allard · 2021-05-06T09:02:38Z

Component: charts, salt

Context:
Actually we use default label selector from kube-prometheus-stack charts release: prometheus-operator and app: prometheus-operator for Prometheus to be able to retrieve ServiceMonitor and PrometheusRules objects.
Those labels do not make real sense and should be replaced by something more meaningful.

Summary:
Replace the default selector by a new one metalk8s.scality.com/monitor: ''.

Acceptance criteria:
Monitoring still works as expected.

Closes: #2199

bert-e · 2021-05-06T09:02:40Z

Hello alexandre-allard-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Status report is not available.

bert-e · 2021-05-06T09:02:46Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/2199-customize-prometheus-label-selector into
development/2.9
w/2.10/improvement/2199-customize-prometheus-label-selector will be merged into development/2.10

The following branches will NOT be impacted:

development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

bert-e · 2021-05-06T09:02:48Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

alexandre-allard · 2021-05-06T09:06:23Z

Putting @scality/storage and @scality/object as reviewers so they are aware of this breaking change.
This change will only be done in metalk8s-2.10.0 and later versions

NicolasT · 2021-05-06T09:55:38Z

For @scality/storage and @scality/object: you can add this new label next to the ones already in place, to be compatible with 'current' MetalK8s as well as 'future' MetalK8s. Then, the current labels can be removed over time, once 'current' MetalK8s is no longer supported (by your application).

TeddyAndrieux · 2021-05-06T13:38:56Z

Worth a changelog entry no ?

The selector is for Probe, ServiceMonitor and PrometheusRule objects and it must match the label `metalk8s.scality.com/monitor: ''` We also add this labels to all resources deployed by the kube-prometheus-stack chart as there is actually no way to only add it to specific resources (except for PrometheusRule). Refs: #2199

We now use custom label selector `metalk8s.scality.com/monitor` instead of the default one `release` and `app` to target ServiceMonitors, PrometheusRules and Probes. kube-prometheus chart render command: ``` ./charts/render.py prometheus-operator \ charts/kube-prometheus-stack.yaml \ charts/kube-prometheus-stack/ \ --namespace metalk8s-monitoring \ --service-config grafana \ metalk8s-grafana-config \ metalk8s/addons/prometheus-operator/config/grafana.yaml \ metalk8s-monitoring \ --service-config prometheus \ metalk8s-prometheus-config \ metalk8s/addons/prometheus-operator/config/prometheus.yaml \ metalk8s-monitoring \ --service-config alertmanager \ metalk8s-alertmanager-config \ metalk8s/addons/prometheus-operator/config/alertmanager.yaml \ metalk8s-monitoring \ --service-config dex \ metalk8s-dex-config \ metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \ --drop-prometheus-rules charts/drop-prometheus-rules.yaml \ > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls ``` Refs: #2199

Refs: #2199

bert-e · 2021-05-07T07:05:14Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

alexandre-allard · 2021-05-07T08:52:40Z

/approve

bert-e · 2021-05-07T08:52:45Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
one peer

Peer approvals must include at least 1 approval from the following list:

The following options are set: approve

TeddyAndrieux · 2021-05-07T08:52:53Z

CHANGELOG.md

@@ -1,18 +1,26 @@
 # CHANGELOG

 ## Release 2.10.0 (in development)
-## Enhancements
+### Enhancements


bert-e · 2021-05-07T08:54:55Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/2.10

The following branches will NOT be impacted:

development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve

bert-e · 2021-05-07T10:15:26Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/2.10

The following branches have NOT changed:

development/2.0
development/2.1
development/2.2
development/2.3
development/2.4
development/2.5
development/2.6
development/2.7
development/2.8
development/2.9

Please check the status of the associated issue None.

Goodbye alexandre-allard-scality.

Oversight from #3356, with this PodMonitor objects are selected using the same label.

Oversight from #3356, with this PodMonitor objects are selected using the same label. Re-render prometheus-operator salt state using: ``` ./charts/render.py prometheus-operator \ charts/kube-prometheus-stack.yaml \ charts/kube-prometheus-stack/ \ --namespace metalk8s-monitoring \ --service-config grafana \ metalk8s-grafana-config \ metalk8s/addons/prometheus-operator/config/grafana.yaml \ metalk8s-monitoring \ --service-config prometheus \ metalk8s-prometheus-config \ metalk8s/addons/prometheus-operator/config/prometheus.yaml \ metalk8s-monitoring \ --service-config alertmanager \ metalk8s-alertmanager-config \ metalk8s/addons/prometheus-operator/config/alertmanager.yaml \ metalk8s-monitoring \ --service-config dex \ metalk8s-dex-config \ metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \ --drop-prometheus-rules charts/drop-prometheus-rules.yaml \ --patch 'PrometheusRule,metalk8s-monitoring,prometheus-operator-kubernetes-system-kubelet,spec:groups:0:rules:1:for,"5m"' \ > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls ```

alexandre-allard requested a review from a team as a code owner May 6, 2021 09:02

alexandre-allard requested review from a team, jonathan-gramain, mrepop and alexandre-merle and removed request for a team May 6, 2021 09:04

alexandre-allard requested review from a team and removed request for jonathan-gramain, alexandre-merle and mrepop May 6, 2021 09:07

alexandre-allard changed the base branch from development/2.9 to development/2.10 May 6, 2021 09:34

alexandre-allard added 5 commits May 7, 2021 09:00

salt, charts: Use custom Prometheus label

948f10b

Refs: #2199

docs: Update PrometheusRule required labels

bf06485

Refs: #2199

Add CHANGELOG entry for new Prometheus label selector

530683c

Refs: #2199

alexandre-allard force-pushed the improvement/2199-customize-prometheus-label-selector branch from f56e8b6 to 530683c Compare May 7, 2021 07:05

TeddyAndrieux approved these changes May 7, 2021

View reviewed changes

CHANGELOG.md

@@ -1,18 +1,26 @@

# CHANGELOG

## Release 2.10.0 (in development)

## Enhancements

### Enhancements

Copy link

Collaborator

TeddyAndrieux May 7, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh 👍

bert-e merged commit 8e827de into development/2.10 May 7, 2021

bert-e deleted the improvement/2199-customize-prometheus-label-selector branch May 7, 2021 10:15

gdemonet added a commit that referenced this pull request Feb 7, 2022

charts,salt: Update PodMonitor selector

79c3eba

Oversight from #3356, with this PodMonitor objects are selected using the same label.

gdemonet mentioned this pull request Feb 7, 2022

charts,salt: Update PodMonitor selector #3692

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Customize Prometheus label selector #3356

Customize Prometheus label selector #3356

alexandre-allard commented May 6, 2021 •

edited

Loading

bert-e commented May 6, 2021

bert-e commented May 6, 2021

bert-e commented May 6, 2021

alexandre-allard commented May 6, 2021 •

edited

Loading

NicolasT commented May 6, 2021

TeddyAndrieux commented May 6, 2021

bert-e commented May 7, 2021

alexandre-allard commented May 7, 2021

bert-e commented May 7, 2021

TeddyAndrieux May 7, 2021

bert-e commented May 7, 2021

bert-e commented May 7, 2021

Customize Prometheus label selector #3356

Customize Prometheus label selector #3356

Conversation

alexandre-allard commented May 6, 2021 • edited Loading

bert-e commented May 6, 2021

Hello alexandre-allard-scality,

bert-e commented May 6, 2021

Integration data created

bert-e commented May 6, 2021

Waiting for approval

alexandre-allard commented May 6, 2021 • edited Loading

NicolasT commented May 6, 2021

TeddyAndrieux commented May 6, 2021

bert-e commented May 7, 2021

Waiting for approval

alexandre-allard commented May 7, 2021

bert-e commented May 7, 2021

Waiting for approval

TeddyAndrieux May 7, 2021

Choose a reason for hiding this comment

bert-e commented May 7, 2021

In the queue

bert-e commented May 7, 2021

alexandre-allard commented May 6, 2021 •

edited

Loading

alexandre-allard commented May 6, 2021 •

edited

Loading