Problems with kubernetes example

[katcipis]

Im having some trouble following the kubernetes example, more specifically on the relabel part:

https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L25-L33

Any of these relabel rules results on the following error on startup:

Jan 12 17:15:05 monitoring docker[6082]: prometheus, version 0.16.1 (branch: stable, revision: 968ee35)
Jan 12 17:15:05 monitoring docker[6082]: build user:       @84057be02ab6
Jan 12 17:15:05 monitoring docker[6082]: build date:       20160111-15:30:09
Jan 12 17:15:05 monitoring docker[6082]: go version:       1.5.1
Jan 12 17:15:05 monitoring systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
Jan 12 17:15:05 monitoring systemd[1]: prometheus.service: Unit entered failed state.
Jan 12 17:15:05 monitoring systemd[1]: prometheus.service: Failed with result 'exit-code'.
Jan 12 17:15:05 monitoring docker[6082]: time="2016-01-12T17:15:05Z" level=info msg="Loading configuration file /prometheus/cfg/prometheus.yml" source="main.go:196"
Jan 12 17:15:05 monitoring docker[6082]: time="2016-01-12T17:15:05Z" level=error msg="Couldn't load configuration (-config.file=/prometheus/cfg/prometheus.yml): relabel configuration requires a regular expression" source="main.go:208"

Im using the latest prometheus image at docker hub. Am I doing something wrong ?

[jimmidyson]

Sorry, looks like we've got examples out of sync with releases - in this case sensible defaults for relabel regexes to reduce verbosity of config. The syntax in the example requires master. You can use prom/prometheus:master & that should work.

[fabxc]

New default values were introduced in master. Please check the same file at
the 0.16.1 tag, which is the version you are running.

The example should be in sync with the code, the latest Docker tag points
to the last release.

On Tue, Jan 12, 2016, 6:26 PM Tiago César Katcipis notifications@github.com
wrote:

Im having some trouble following the kubernetes example, more specifically
on the relabel part:

https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L25-L33

Any of these relabel rules results on the following error on startup:

Jan 12 17:15:05 monitoring docker[6082]: prometheus, version 0.16.1 (branch: stable, revision: 968ee35)
Jan 12 17:15:05 monitoring docker[6082]: build user: @84057be02ab6
Jan 12 17:15:05 monitoring docker[6082]: build date: 20160111-15:30:09
Jan 12 17:15:05 monitoring docker[6082]: go version: 1.5.1
Jan 12 17:15:05 monitoring systemd[1]: prometheus.service: Main process exited, code=exited, status=1/FAILURE
Jan 12 17:15:05 monitoring systemd[1]: prometheus.service: Unit entered failed state.
Jan 12 17:15:05 monitoring systemd[1]: prometheus.service: Failed with result 'exit-code'.
Jan 12 17:15:05 monitoring docker[6082]: time="2016-01-12T17:15:05Z" level=info msg="Loading configuration file /prometheus/cfg/prometheus.yml" source="main.go:196"
Jan 12 17:15:05 monitoring docker[6082]: time="2016-01-12T17:15:05Z" level=error msg="Couldn't load configuration (-config.file=/prometheus/cfg/prometheus.yml): relabel configuration requires a regular expression" source="main.go:208"

Im using the latest prometheus image at docker hub. Am I doing something
wrong ?

—
Reply to this email directly or view it on GitHub
#1312.

[katcipis]

Using the prometheus:master image made everything work fine, thanks @jimmidyson. Now im having other problem.

Im trying to scrap information from my RabbitMQ instance, that is running inside k8s. I added the exporter on the same pod as the rabbit and exported the container port.

rc file:

apiVersion: v1
kind: ReplicationController
metadata:
  name: rabbitmq
spec:
  replicas: 1
  selector:
    app: rabbitmq
  template:
    metadata:
      name: rabbitmq
      labels:
        app: rabbitmq
    spec:
      containers:
      - image: rabbitmq:3.5.4-management
        name: rabbitmq
        volumeMounts:
        - mountPath: /var/lib/rabbitmq
          name: rabbitmq-vol
        ports:
        - containerPort: 5672
          name: service
        - containerPort: 15672
          name: management
      - image: kbudde/rabbitmq-exporter
        name: rabbitmq-exporter
        volumeMounts:
        - mountPath: /var/lib/rabbitmq-exporter
          name: rabbitmq-vol
        ports:
        - containerPort: 9090
          name: exporter
      volumes:
        - name: rabbitmq-vol
          persistentVolumeClaim:
           claimName: rabbitmq

svc file:

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  labels:
    name: rabbitmq
  annotations:
    prometheus.io/scrape: "true"
spec:
  ports:
    - port: 5672
      name: service
      targetPort: service
      protocol: TCP
    - port: 9090
      name: exporter
      targetPort: exporter
      protocol: TCP
  # just like the selector in the replication controller,
  # but this time it identifies the set of pods to load balance
  # traffic to.
  selector:
    app: rabbitmq

Then I check the running service, the endpoint is there:

Name:           rabbitmq
Namespace:      default
Labels:         name=rabbitmq
Selector:       app=rabbitmq
Type:           ClusterIP
IP:         10.100.200.148
Port:           service 5672/TCP
Endpoints:      10.244.50.21:5672
Port:           exporter    9090/TCP
Endpoints:      10.244.50.21:9090
Session Affinity:   None
No events.

Using the default configuration, found on the docs, I got no data (and no warning or error on the logs). If I configure prometheus to manually scrap the endpoint 10.244.50.21:9090, it works, so it is not connectivity.

I thought I could have missed the annotation, but when I tried to create it I got the error:

root@195da9ce6e6b:/harbor/data# kubectl -s http://10.4.2.134:8080 annotate svc rabbitmq prometheus.io/scrape="true"
error: --overwrite is false but found the following declared annotation(s): 'prometheus.io/scrape' already has a value (true)

So my rabbitmq svc is already annotated. Here is my whole config ( is where the address of the master would be, im working with ip right now, not DNS):

      global:
        scrape_interval:     15s
        evaluation_interval: 15s
        external_labels:
          monitor: 'core'
      scrape_configs:
        - job_name: 'prometheus'
          scrape_interval: 5s
          scrape_timeout: 10s
          target_groups:
            - targets: ['127.0.0.1:9100']
        - job_name: 'k8s-cluster'
          scrape_interval: 5s
          scrape_timeout: 10s
          kubernetes_sd_configs:
          - api_servers:
            - 'http://<master-ip>:8080'
            bearer_token_file: bearer_token
            kubelet_port: 10255
            in_cluster: true
          relabel_configs:
          - source_labels: [__meta_kubernetes_role]
            action: keep
            regex: (?:apiserver|node)
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - source_labels: [__meta_kubernetes_role]
            action: replace
            target_label: kubernetes_role
        # Scrape config for service endpoints.
        #
        # The relabeling allows the actual service scrape endpoint to be configured
        # via the following annotations:
        #
        # * `prometheus.io/scrape`: Only scrape services that have a value of `true`
        # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
        # to set this to `https` & most likely set the `tls_config` of the scrape config.
        # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
        # * `prometheus.io/port`: If the metrics are exposed on a different port to the
        # service then set this appropriately.
        #
        - job_name: 'kubernetes-service-endpoints'
          kubernetes_sd_configs:
          - api_servers:
            - 'http://<master-ip>:8080'
            bearer_token_file: bearer_token
            kubelet_port: 10255
            in_cluster: true
          relabel_configs:
          - source_labels: [__meta_kubernetes_role, __meta_kubernetes_service_annotation_prometheus_io_scrape]
            action: keep
            regex: endpoint;true
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
            action: replace
            target_label: __scheme__
            regex: (https?)
          - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
          - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: (.+)(?::\d+);(\d+)
            replacement: $1:$2
          - action: labelmap
            regex: __meta_kubernetes_service_label_(.+)
          - source_labels: [__meta_kubernetes_role]
            action: replace
            target_label: kubernetes_role
          - source_labels: [__meta_kubernetes_service_namespace]
            action: replace
            target_label: kubernetes_namespace
          - source_labels: [__meta_kubernetes_service_name]
            action: replace
            target_label: kubernetes_name

Any idea of what I may be doing wrong ?

[jimmidyson]

Does the pod show up as a target in the Prometheus status page? You may need to annotate the service with the port number to use like: prometheus.io/port: "9090".

[katcipis]

@jimmidyson I was pretty stupid, taking a look on the status page made the problem quite clear. It is trying to scrap rabbitmq on the 5672 port, which is the first exported port.

I thought the default port was 9090, but it seems to be the first exported one (on k8s, if you dont mention the port name it is an ambiguity, since there is multiple ports exported on the service).

[jimmidyson]

Ah OK! So the prometheus.io/port annotation should work for you.

[katcipis]

It worked, but by default it tried to scrap on "/" instead of "/metrics". Shouldn't "/metrics" be the default ?

[katcipis]

It worked, thanks @jimmidyson :-) (after I manually defined the path to be "/metrics").

I can add an example on the pod + service description to make RabbitMQ work on this setup. If it seems to be a good idea, where is the best place for me to add it on the project ?

[jimmidyson]

It certainly should default to /metrics - I'll fix that now.

Prometheus doesn't really have external example setup so not sure where best to put it. @fabxc @brian-brazil Any thoughts? Seems useful to me, using RabbitMQ as an example to show the configuration of the Kubernetes services required for discovery.

[sdurrheimer]

@jimmidyson Maybe in a contrib folder.

[jimmidyson]

@sdurrheimer Nice idea.

@katcipis PR welcome I'm sure.

[jimmidyson]

@katcipis The problem with the scrape path was just a small change to example relabeling - https://github.com/prometheus/prometheus/pull/1313/files

[katcipis]

@jimmidyson just tested here, worked like a charm ;-).

I noticed that when I omit the relabeling prometheus tries to scrap everything from k8s (all pods, assuming the default DNS name of them).

Is this expected ?

[beorn7]

To quote @jimmidyson : "You can do this via relabelling. See https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L53-L56 for an example. What this is doing is only keeping Kubernetes services that are annotated with prometheus.io/scrape=true. I find it works best to whitelist services in this way."

Does that help?

I assume this issue is solved. Please re-open with more information if not.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.