[prometheus-kube-stack] Cannot get metrics from services in other namespaces

[agtogna]

Describe the bug a clear and concise description of what the bug is.

Since version 38.x.y metrics from services in namespaces other than Prometheus' are not scraped anymore. Since the most important are the kubelet metrics I'll refer to them.
The ServiceMonitor is the same as in version 37.x, with a correct namespace selector:
namespaceSelector: matchNames: - kube-system selector: matchLabels: app.kubernetes.io/name: kubelet k8s-app: kubelet

What's your helm version?

version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.6", GitCommit:"42a9a90338d705a1650fb68b7891f84b62adb0b0", GitTreeState:"clean", BuildDate:"2022-06-15T04:25:21Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

kube-prometheus-stack

What's the chart version?

39.0.0

What happened?

Since version 38 of the kube-prometheus-stack chart, services in namespaces other than Prometheus' one are not scraped anymore.

What you expected to happen?

No response

How to reproduce it?

Either:

• upgrade from chart 37.x.y to 38.x.y or 39.x.y
• install chart 38.x.y or 39.x.y

Enter the changed values of values.yaml?

prometheus:
  enabled: true
  prometheusSpec:
    replicas: 1
    shards: 1
    logLevel: info
    logFormat: json
    podMonitorSelectorNilUsesHelmValues: false
    podMonitorNamespaceSelector:
      any: true
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorNamespaceSelector:
      any: true
    probeSelectorNilUsesHelmValues: false

Enter the command that you execute and failing/misfunctioning.

helm upgrade prometheus prometheus-community/kube-prometheus-stack --version 39.0.0 -n <prometheus_namespace> -f values.yaml

Anything else we need to know?

Since version 38.x you cannot use

    podMonitorNamespaceSelector:
       any: true
    serviceMonitorNamespaceSelector:
       any: true

but instead:
ignoreNamespaceSelectors: true

[arpitjindal97]

The options serviceMonitorSelectorNilUsesHelmValues & podMonitorSelectorNilUsesHelmValues only applies to serviceMonitorSelector & podMonitorSelector respectively but not on namespaces.

Try the configuration below, It will be able to capture all the ServiceMonitors from all namespaces

prometheus:
  service:
    type: NodePort
  prometheusSpec:
    scrape_interval: 10s
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorNamespaceSelector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: Exists
        values:

Do understand that, you still need to create ServiceMonitor resource for your services which you want to monitor in their namespaces.

Sample apiserver-servicemonitor.yaml:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: product-apiserver
  namespace: default
spec:
  endpoints:
    - honorLabels: true
      path: /metrics
      port: http-web
      scheme: http
      scrapeTimeout: 10s
  selector:
    matchLabels:
      app: apiserver

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[antinova]

@arpitjindal97 I wasn't able to get your config to work. It was throwing an error about the empty values key. However, removing the values key made it work:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false
    serviceMonitorNamespaceSelector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: Exists