Update manifests to latest version of Prometheus Operator

[brancz]

Tried out with a container image of HEAD of the Prometheus Operator.

List of changes divided into commits:

• Set lower resource requests by default (simply for demo purposes, as this currently won't work on minikube for example with only 1Gb of memory)
• Remove unnecessary prometheus.io/scrape annotation: (tl;dr) more confusing then helpful
• Add Alertmanager support in TPRs and manifests themselves (recently added to Prometheus Operator)
• Add kube-dns service for skydns and dnsmasq metrics to be collected. This change reflects work upstream and on bootkube.
• Update node-exporter version

It would probably make sense to cut another release of the Prometheus Operator, just so we can at least partially version this until we get versioned user-guides.

@fabxc

[fabxc]

👍 looks good.

Yes, we should cut another version to integrate into Tectonic etc.

[fabxc]

The annotation is not supported by the operator. ServiceMonitors should be used instead. The annotation is very limited by its nature and there's no feasible way to support anything beyond "scrape on" and a single port. A per-service scrape interval, multi-port pods, authentication, honorLabels, and more are all not possible. It also provides no meaningful way to assign services to be scraped by different Prometheus instances. ServiceMonitors allow all these without exposing any complex relabelling and there's really no practical reason for sticking with the annotation.

…

On Fri, Jun 2, 2017 at 2:26 PM Julian V. Modesto ***@***.***> wrote: @brancz <https://github.com/brancz> sorry if this isn't the best place to ask! Regarding 0c1bb5c <0c1bb5c>, why were prometheus.io/scrape annotations removed? Is there any way to configure Prometheus Operator to discover pods to scrape based on these annotations still? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#16 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuA8ncH_YnOGepPiKJVVzmgcMQwTWd5ks5sAH3vgaJpZM4LCl_q> .

[julianvmodesto]

Got it – this is helpful and makes sense after consuming, thank you!

[cornelius-keller]

The problem I see here is that this annotations are fully self service and a well working way to communicate the metrics that should be scraped to prometheus. With ServiceMonitors this self service is not working anymore as they need to be in the monitoring namespace and I don't want to give everybody access to it.
As discussed here: prometheus-operator/prometheus-operator#813 .
So for me regarding ease of setup etc. prometheus-operator is a great step, regarding the self service kapabilities that my previeous setup had it is a step back.

[brancz]

The ServiceMonitor has a namespaceselector for the services, so the only thing you to do is have a convention in your organization, which boils down to the same as the annotation. You simply need you organization to have a commonly named port and a label like "monitoring: true". The namespace selector can then select Services globally.

[brancz]

However, we acknowledge the want that people have, and are trying to think of a way we can introduce something more broad without violating the concerns we currently have.

[StevenACoffman]

We have an established convention, but also often have third party tools installed which have their own conflicting conventions. As a hybrid solution, we are applying a custom prometheus on top of the operator. Thanks again @solsson

[maver1ck]

I'd like to add one more thing to this issue.
There is a lot of 3rd party tools and ready to use helm charts that are using annotations.
And we can't use it out of the box.

Example:
https://github.com/confluentinc/cp-helm-charts

[StevenACoffman]

That is exactly our issue. 3rd party helm apps often have this need, which is in conflict with our using the operator at present.

[maver1ck]

I found following solution.
In prometheus helm chart configuration add this:

  additionalScrapeConfigs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

Then all pods with prometheus.io/scrape annotations are processed.
You don't need custom prometheus for that :)

[suppix]

@maver1ck Could you please tell me where I should put this configuration ?

[maver1ck]

You need to add this to helm chart configuration (values.yaml)
https://github.com/coreos/prometheus-operator/blob/master/helm/prometheus/values.yaml#L353

Unfortunately this option is undocumented.

[absolutejam]

I found following solution.
In prometheus helm chart configuration add this:

  additionalScrapeConfigs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

Then all pods with prometheus.io/scrape annotations are processed.
You don't need custom prometheus for that :)

There is also some information available at https://github.com/coreos/prometheus-operator/blob/master/Documentation/additional-scrape-config.md

[jmvizcainoio]

you can also add this config to scrape services

      - job_name: 'kubernetes-service'
        kubernetes_sd_configs:
        - role: service
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_service_name

[JulienBreux]

Update about values.yaml location; https://github.com/helm/charts/blob/master/stable/prometheus-operator/values.yaml

[bchanson]

If you're compiling the yamls using the jsonnet file. Follow the instructions mentioned earlier to create the additional scrape config secret.

I tried to patch the prometheus resource directly after building/applying the yamls via jsonnet. It didn't work. I had to add the additionalScrapeConfigs config to the jsonnet file.

This does not go in the _config+:: section. It goes in prometheus+:: like the serviceMonitor definitions.

    prometheus+:: {
      prometheus+: {
        spec+: {
          additionalScrapeConfigs+: {
            name: 'additional-scrape-configs',
            key: 'additonal-scrape-configs.yaml'
          },
        },
      },
    },