podMonitorSelector alone does not work

[matthiasr]

What happened?

When using a Prometheus that only has a podMonitorSelector: {} but not a serviceMonitorSelector: {}, the pod monitor is never synced.

When starting this way, no targets appear in Prometheus. Since these are the only ones in the namespace, the secret/prometheus-prometheus entry prometheus.yaml.gz remains an empty string.

Adding a dummy serviceMonitorSelector: {} fixes this, even though there are no service monitors anywhere in the cluster.

Conversely, when updating the Prometheus to remove the serviceMonitorSelector, updates to the podMonitor are no longer reflected in Prometheus.

Did you expect to see something different?

It should be possible to use only PodMonitor.

How to reproduce it (as minimally and precisely as possible):

Apply the following manifests (or a variation on them) plus a deployment/pods with matching labels.

rocket-podmonitor.yaml.gz
rocket-prometheus.yaml.gz

Observe that kubectl get -n rocket secret/prometheus-prometheus -o yaml is empty and the timestamp is from before submitting the podmonitor:

secret.yaml

apiVersion: v1
data:
  prometheus.yaml.gz: ""
kind: Secret
metadata:
  annotations:
    empty: "true"
  creationTimestamp: "2020-04-24T08:25:06Z"
  labels:
    managed-by: prometheus-operator
  name: prometheus-prometheus
  namespace: rocket
  ownerReferences:
  - apiVersion: monitoring.coreos.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: Prometheus
    name: prometheus
    uid: 0e359730-10f3-405d-b4fd-5528cc8c1bde
  resourceVersion: "69659848"
  selfLink: /api/v1/namespaces/rocket/secrets/prometheus-prometheus
  uid: a8de2009-0c40-4c89-9a8c-3f273cb8f4fb
type: Opaque

Delete the podmonitor. Update the Prometheus object to add serviceMonitorSelector: {}. Add the podmonitor. Observe that the secret now contains information, and the debug log output is much more comprehensive.

Now the secret changes and targets appear.

To test the other way around, modify the Prometheus to remove the serviceMonitorSelector. Delete the PodMonitor. Observe that the secret does not change, and the targets are still being targetted.

Environment

• Prometheus Operator version: v0.38.1

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a17149e1a189050796ced469dbd78d380f2ed5ef", GitTreeState:"clean", BuildDate:"2020-04-16T11:44:51Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a17149e1a189050796ced469dbd78d380f2ed5ef", GitTreeState:"clean", BuildDate:"2020-04-16T11:36:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind: custom/in-house
• Manifests: see attachments
• Prometheus Operator Logs:

podmonitor sync log without servicemonitorselector

2020-04-24T08:26:48.582083929	stdout	level=debug ts=2020-04-24T08:26:48.581811457Z caller=operator.go:734 component=prometheusoperator msg="PodMonitor added"
2020-04-24T08:26:48.582144179	stdout	level=info ts=2020-04-24T08:26:48.581932587Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=rocket/prometheus
2020-04-24T08:26:48.582155033	stdout	level=debug ts=2020-04-24T08:26:48.581959681Z caller=rules.go:158 component=prometheusoperator msg="selected RuleNamespaces" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:26:48.582166795	stdout	level=debug ts=2020-04-24T08:26:48.582054054Z caller=rules.go:200 component=prometheusoperator msg="selected Rules" rules= namespace=rocket prometheus=prometheus
2020-04-24T08:26:48.587282574	stdout	level=debug ts=2020-04-24T08:26:48.587147904Z caller=rules.go:75 component=prometheusoperator msg="no PrometheusRule changes" namespace=rocket prometheus=prometheus
2020-04-24T08:26:48.591759897	stdout	level=debug ts=2020-04-24T08:26:48.5915708Z caller=operator.go:1776 component=prometheusoperator msg="filtering namespaces to select ServiceMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:26:48.591801483	stdout	level=debug ts=2020-04-24T08:26:48.591625353Z caller=operator.go:1810 component=prometheusoperator msg="selected ServiceMonitors" servicemonitors= namespace=rocket prometheus=prometheus
2020-04-24T08:26:48.600973146	stdout	level=debug ts=2020-04-24T08:26:48.600832446Z caller=operator.go:1741 component=prometheusoperator msg="updated tlsAssetsSecret" secretname=prometheus-prometheus-tls-assets
2020-04-24T08:26:48.618293342	stdout	level=debug ts=2020-04-24T08:26:48.618114567Z caller=operator.go:1169 component=prometheusoperator msg="new statefulset generation inputs match current, skipping any actions"

podmonitor sync log with servicemonitorselector

2020-04-24T08:30:11.14453533	stdout	level=debug ts=2020-04-24T08:30:11.144222353Z caller=operator.go:734 component=prometheusoperator msg="PodMonitor added"
2020-04-24T08:30:11.144585071	stdout	level=info ts=2020-04-24T08:30:11.144365614Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=rocket/prometheus
2020-04-24T08:30:11.144598679	stdout	level=debug ts=2020-04-24T08:30:11.144402382Z caller=rules.go:158 component=prometheusoperator msg="selected RuleNamespaces" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.144610033	stdout	level=debug ts=2020-04-24T08:30:11.144438181Z caller=rules.go:200 component=prometheusoperator msg="selected Rules" rules= namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.151218384	stdout	level=debug ts=2020-04-24T08:30:11.150874744Z caller=rules.go:75 component=prometheusoperator msg="no PrometheusRule changes" namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.151256577	stdout	level=debug ts=2020-04-24T08:30:11.150961868Z caller=operator.go:1776 component=prometheusoperator msg="filtering namespaces to select ServiceMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.151272637	stdout	level=debug ts=2020-04-24T08:30:11.151042124Z caller=operator.go:1810 component=prometheusoperator msg="selected ServiceMonitors" servicemonitors= namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.151283736	stdout	level=debug ts=2020-04-24T08:30:11.151073302Z caller=operator.go:1840 component=prometheusoperator msg="filtering namespaces to select PodMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.151388371	stdout	level=debug ts=2020-04-24T08:30:11.151105777Z caller=operator.go:1853 component=prometheusoperator msg="selected PodMonitors" podmonitors=rocket/rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.161618215	stdout	level=debug ts=2020-04-24T08:30:11.161381006Z caller=operator.go:1680 component=prometheusoperator msg="current Prometheus configuration has changed"
2020-04-24T08:30:11.161698038	stdout	level=debug ts=2020-04-24T08:30:11.161442325Z caller=operator.go:1685 component=prometheusoperator msg="updating Prometheus configuration secret"
2020-04-24T08:30:11.167919342	stdout	level=debug ts=2020-04-24T08:30:11.167710977Z caller=operator.go:821 component=prometheusoperator msg="Secret updated"
2020-04-24T08:30:11.171925662	stdout	level=debug ts=2020-04-24T08:30:11.171726295Z caller=operator.go:1776 component=prometheusoperator msg="filtering namespaces to select ServiceMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.171962328	stdout	level=debug ts=2020-04-24T08:30:11.171782069Z caller=operator.go:1810 component=prometheusoperator msg="selected ServiceMonitors" servicemonitors= namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.181964465	stdout	level=debug ts=2020-04-24T08:30:11.181758411Z caller=operator.go:1741 component=prometheusoperator msg="updated tlsAssetsSecret" secretname=prometheus-prometheus-tls-assets
2020-04-24T08:30:11.194976071	stdout	level=debug ts=2020-04-24T08:30:11.19473885Z caller=operator.go:1169 component=prometheusoperator msg="new statefulset generation inputs match current, skipping any actions"
2020-04-24T08:30:11.195047161	stdout	level=info ts=2020-04-24T08:30:11.194813741Z caller=operator.go:1094 component=prometheusoperator msg="sync prometheus" key=rocket/prometheus
2020-04-24T08:30:11.195061312	stdout	level=debug ts=2020-04-24T08:30:11.194843327Z caller=rules.go:158 component=prometheusoperator msg="selected RuleNamespaces" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.195071835	stdout	level=debug ts=2020-04-24T08:30:11.19487092Z caller=rules.go:200 component=prometheusoperator msg="selected Rules" rules= namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.224843498	stdout	level=debug ts=2020-04-24T08:30:11.224591658Z caller=rules.go:75 component=prometheusoperator msg="no PrometheusRule changes" namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.224880675	stdout	level=debug ts=2020-04-24T08:30:11.224642386Z caller=operator.go:1776 component=prometheusoperator msg="filtering namespaces to select ServiceMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.224894124	stdout	level=debug ts=2020-04-24T08:30:11.224670865Z caller=operator.go:1810 component=prometheusoperator msg="selected ServiceMonitors" servicemonitors= namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.22490586	stdout	level=debug ts=2020-04-24T08:30:11.224694141Z caller=operator.go:1840 component=prometheusoperator msg="filtering namespaces to select PodMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.224917933	stdout	level=debug ts=2020-04-24T08:30:11.224721452Z caller=operator.go:1853 component=prometheusoperator msg="selected PodMonitors" podmonitors=rocket/rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.234619225	stdout	level=debug ts=2020-04-24T08:30:11.23436524Z caller=operator.go:1677 component=prometheusoperator msg="updating Prometheus configuration secret skipped, no configuration change"
2020-04-24T08:30:11.239078369	stdout	level=debug ts=2020-04-24T08:30:11.238847652Z caller=operator.go:1776 component=prometheusoperator msg="filtering namespaces to select ServiceMonitors from" namespaces=rocket namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.239113264	stdout	level=debug ts=2020-04-24T08:30:11.238900354Z caller=operator.go:1810 component=prometheusoperator msg="selected ServiceMonitors" servicemonitors= namespace=rocket prometheus=prometheus
2020-04-24T08:30:11.248703325	stdout	level=debug ts=2020-04-24T08:30:11.248528895Z caller=operator.go:1741 component=prometheusoperator msg="updated tlsAssetsSecret" secretname=prometheus-prometheus-tls-assets
2020-04-24T08:30:11.261782267	stdout	level=debug ts=2020-04-24T08:30:11.26155147Z caller=operator.go:1169 component=prometheusoperator msg="new statefulset generation inputs match current, skipping any actions"

Anything else we need to know?:

Adding a service monitor selector is a viable workaround for me.

[brancz]

This is unfortunately due to some historic legacy. I personally think we should actually remove this, since we have the additionalScrapeConfigs feature. (this used to be intentional to allow the user to manage the prometheus config themselves if service monitor selectors were unspecified, the additionalScrapeConfigs feature turned out to be the better choice though)

If you want to do a PR to fix this, I'd be happy to move forward with it.

[matthiasr]

Hmm, that sounds like it would be a breaking change for someone. I think I can see how to extend the existing logic to be "if service and pod monitors were unspecified", what do you think about doing that first, and then tackling the legacy-cleanup separately?

…

On Fri, Apr 24, 2020 at 5:17 PM Frederic Branczyk ***@***.***> wrote: This is unfortunately due to some historic legacy. I personally think we should actually remove this, since we have the additionalScrapeConfigs feature. (this used to be intentional to allow the user to manage the prometheus config themselves if service monitor selectors were unspecified, the additionalScrapeConfigs feature turned out to be the better choice though) If you want to do a PR to fix this, I'd be happy to move forward with it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3164 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABAEBRFU723YSAQCUT2QKLROHCR7ANCNFSM4MP44VEA> .

[brancz]

That sounds good to me! Maybe we can specifically call this out in field descriptions (and change log) that this behavior will be dropped in the future and we do so in a couple of releases?