serviceMonitorNamespaceSelector and serviceMonitorSelector don't work

[jescarri]

What did you do?

Deployed a prometheus with serviceMonitorNamespaceSelector and serviceMonitorSelector

What did you expect to see?

Scrape jobs for the matching namespace / serviceMonitor

What did you see instead? Under which circumstances?

Only one scrape job.

Environment

• Kubernetes version information:

  insert output of kubectl version here
  Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.9+coreos.1", GitCommit:"cd373fe93e046b0a0bc7e4045af1bf4171cea395", GitTreeState:"clean", BuildDate:"2018-03-13T21:28:21Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

  Tectonic instaler

• Manifests:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app: prometheus
    chart: prometheus-2.0.0
    environment: nonprod
    heritage: Tiller
    inventorycode: ""
    productcode: foo
    prometheus: jesus1
    release: jesus1
  name: jesus1
  namespace: foo
spec:
  additionalScrapeConfigs:
    key: scrape-configs.yaml
    name: jesus1-scrape-configs
  alerting:
    alertmanagers:
    - name: alertmanager-jesus1
      namespace: foo
      port: http
  evaluationInterval: 15s
  externalLabels:
    src_prometheus: foo-us-east-1
  externalUrl: http://prometheus.jesus2.sandbox3.us-east-1.test.io
  podMetadata:
    labels:
      tier: cs
  replicas: 1
  resources:
    limits:
      cpu: 500m
      memory: 3000Mi
    requests:
      cpu: 500m
      memory: 3000Mi
  retention: 12h
  ruleSelector:
    matchLabels:
      prometheus: jesus1
      role: alert-rules
  scrapeInterval: 60s
  serviceMonitorNamespaceSelector:
    matchNames:
    - foo
    - bar
  serviceMonitorSelector:
    matchExpressions:
    - key: service_monitor
      operator: In
      values:
      - jesus1
      - jesus2
  version: v2.2.1

• Prometheus Operator Logs:

No errors.

Service monitors exist as follow:

Namespace foo:

kubectl -n foo get servicemonitors -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    creationTimestamp: 2018-05-10T22:58:06Z
    deletionGracePeriodSeconds: null
    deletionTimestamp: null
    initializers: null
    labels:
      app: prometheus
      chart: prometheus-2.0.0
      environment: nonprod
      heritage: Tiller
      inventorycode: ""
      productcode: foo
      prometheus: scrape
      release: jesus1
      service_monitor: jesus1
    name: jesus1
    namespace: foo
    resourceVersion: "33261069"
  spec:
    endpoints:
    - port: http
    selector:
      matchLabels:
        release: jesus1
    targetLabels:
    - app
    - prometheus
    - ticketmaster.com/productcode
    - ticketmaster.com/inventorycode
    - ticketmaster.com/environment
    - name
    - environment
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Namespace bar:

apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    creationTimestamp: 2018-05-10T23:20:48Z
    deletionGracePeriodSeconds: null
    deletionTimestamp: null
    initializers: null
    labels:
      app: prometheus
      chart: prometheus-2.0.0
      environment: nonprod
      heritage: Tiller
      inventorycode: ""
      productcode: bar
      prometheus: scrape
      release: jesus2
      service_monitor: jesus2
    name: jesus2
    namespace: bar
    resourceVersion: "33272916"
  spec:
    endpoints:
    - port: http
    selector:
      matchLabels:
        release: jesus2
    targetLabels:
    - app
    - prometheus
    - ticketmaster.com/productcode
    - ticketmaster.com/inventorycode
    - ticketmaster.com/environment
    - name
    - environment
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Bot service monitors match healthy services.

However in prometheus jesus1 it only creates a matching scrape config for service monitor jesus1

global:
  scrape_interval: 1m
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    prometheus: foo/jesus1
    prometheus_replica: prometheus-jesus1-0
    src_prometheus: foo--us-east-1
alerting:
  alert_relabel_configs:
  - separator: ;
    regex: prometheus_replica
    replacement: $1
    action: labeldrop
  alertmanagers:
  - kubernetes_sd_configs:
    - api_server: null
      role: endpoints
      namespaces:
        names:
        - foo
    scheme: http
    path_prefix: /
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager-jesus1
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      separator: ;
      regex: http
      replacement: $1
      action: keep
scrape_configs:
- job_name: foo/jesus1/0
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - api_server: null
    role: endpoints
    namespaces:
      names:
      - foo
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_label_release]
    separator: ;
    regex: jesus1
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: http
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app]
    separator: ;
    regex: (.+)
    target_label: app
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_prometheus]
    separator: ;
    regex: (.+)
    target_label: prometheus
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_ticketmaster_com_productcode]
    separator: ;
    regex: (.+)
    target_label: ticketmaster_com_productcode
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_ticketmaster_com_inventorycode]
    separator: ;
    regex: (.+)
    target_label: ticketmaster_com_inventorycode
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_ticketmaster_com_environment]
    separator: ;
    regex: (.+)
    target_label: ticketmaster_com_environment
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_name]
    separator: ;
    regex: (.+)
    target_label: name
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_label_environment]
    separator: ;
    regex: (.+)
    target_label: environment
    replacement: ${1}
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: job
    replacement: ${1}
    action: replace
  - separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: http
    action: replace

[brancz]

Which version of the Prometheus operator are you using? The serviceMonitorNamespaceSelector was only introduced in the latest (v0.19.0) release.

[jescarri]

@brancz Do'h! I forgot to mention it, my bad, I'm using v0.19.0

[mxinden]

@jescarri Thanks a lot for the detailed bug report.

Your Prometheus CRD has a wrong ServiceMonitorNamespaceSelector:

  serviceMonitorNamespaceSelector:
    matchNames:
    - foo
    - bar

ServiceMonitorNamespaceSelector is a k8s.io/apimachinery/pkg/apis/meta/v1.LabelSelector, not a github.com/coreos/prometheus-operator/pkg/client/monitoring/v1.NamespaceSelector.

I would suggest adding a label to each namespace that you want to scan for ServiceMonitors (e.g. prometheus: front-end-team) for a particular Prometheus (e.g. the front-end-team Prometheus).

Then in your Prometheus CRD you would write:

  serviceMonitorNamespaceSelector:
    matchLabels:
      prometheus: front-end-team

So now you might be asking yourself: "Why did this work with jesus1 and not jesus2?"

Kubernetes ignores unknown fields. In this case it ignored the matchNames key. Thereby the serviceMonitorNamespaceSelector was nil. By default, if the serviceMonitorNamespaceSelector is nil the Prometheus Operator only scans the namespace of the Prometheus CRD for ServiceMonitors. Both the Prometheus CRD as well as the jesus1 ServiceMonitor are inside the foo namespace.

Let me know if this helps.

[jescarri]

Ha!, that made the difference, I ended up using a matchExpression.

Thanks for the help!.

[faheem-nadeem]

@mxinden Have a question. I am not sure we are able to attach labels to namespaces created by Helm. Currently we have all our stack in helm. Can you enlighten me if k8s.io/apimachinery/pkg/apis/meta/v1.LabelSelector is able to select let say metadata name rather than labels.

[mxinden]

@faheem-cliqz as far as I know you can only select labels via the LabelSelector, which sounds like a clean design to me.

Maybe @gianrubio can give you more advice on how to add namespace labels via helm.

@faheem-cliqz This sounds like a new question to me. In the future, please create a new Github issue for it.

[faheem-nadeem]

Found a lingering feature request in helm for this. Referred it. Moving on will create a new issue. May be @gianrubio knows of any hacks for now to get this.

[stealthybox]

By default, if the serviceMonitorNamespaceSelector is nil the Prometheus Operator only scans the namespace of the Prometheus CRD for ServiceMonitors.

if anybody is curious, this is the code that causes this behavior (pinned @ ref a36a34f)

https://github.com/coreos/prometheus-operator/blob/a36a34f/pkg/prometheus/operator.go#L1335-L1348

[karasjoh000]

You can add labels to the namespaces and then reference those labels. If you want to use the namespace names, Kubernetes makes a label foreach namespace with its own name, kubernetes.io/metadata.name: <namespace-name>, by default:

  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:   
        - monitoring
        - prometheus
        - default

[trainingUI]

You can add labels to the namespaces and then reference those labels. If you want to use the namespace names, Kubernetes makes a label foreach namespace with its own name, kubernetes.io/metadata.name: <namespace-name>, by default:

  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: In
      values:   
        - monitoring
        - prometheus
        - default

not working for default labels

[karasjoh000]

@trainingUI what Kubernetes version are you using? For older versions of k8s they don't have kubernetes.io/metadata.name label. Do kubectl get ns/<namespace> -o json to see what labels it has (.metadata.labels). If it does not have any labels add your own with:

for ns in monitoring prometheus default; do 
    kubectl patch ns/$ns --patch "{ \"metadata\": { \"labels\": { \"name\": \"$ns\" } } }"
done 

which you can reference with:

  serviceMonitorNamespaceSelector: 
    matchExpressions:
      - key: name
        operator: In
        values:   
          - monitoring
          - prometheus
          - default

Make sure you include the service monitor labels also:

  serviceMonitorSelector: 
    matchExpressions:
      - key: k8s-app
        operator: In
        values:   
          - registry

[trainingUI]

kubectl patch ns/$ns --patch "{ \"metadata\": { \"labels\": { \"name\": \"$ns\" } } }"

It worked, Thanks!