prometheus configuration for kubernetes from example folder doesn't work with 1.3

[vorozhko]

What did you do?
Upgraded prometheus to 1.3 in our GKE cluster.

What did you expect to see?
Working prometheus.

What did you see instead? Under which circumstances?
First I have seen:
time="2016-11-02T13:55:23Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): Unknown Kubernetes SD role "apiserver"" source="main.go:149"

After I deleted the whole job-name with apiserver role I got following:
time="2016-11-02T14:24:20Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in kubernetes_sd_config: in_cluster, api_servers" source="main.go:149

Environment
Prometheus container v1.3.0 in GKE cluster.

• System information:

• Prometheus version:

  1.3

• Prometheus configuration file:

scrape_configs:
    - job_name: 'kubernetes-nodes'

      # Default to scraping over https. If required, just disable this or change to
      # `http`.
      scheme: https

      # This TLS & bearer token file config is used to connect to the actual scrape
      # endpoints for cluster components. This is separate to discovery auth
      # configuration (`in_cluster` below) because discovery & scraping are two
      # separate concerns in Prometheus.
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        # If your node certificates are self-signed or use a different CA to the
        # master CA, then disable certificate verification below. Note that
        # certificate verification is an integral part of a secure infrastructure
        # so this should only be disabled in a controlled environment. You can
        # disable certificate verification by uncommenting the line below.
        #
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - api_servers:
        - 'https://kubernetes.default.svc'
        in_cluster: true
        role: node

      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    # Scrape config for service endpoints.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape services that have a value of `true`
    # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
    # to set this to `https` & most likely set the `tls_config` of the scrape config.
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
    - job_name: 'kubernetes-service-endpoints'

      kubernetes_sd_configs:
      - api_servers:
        - 'https://kubernetes.default.svc'
        in_cluster: true
        role: endpoint

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    # Example scrape config for probing services via the Blackbox Exporter.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/probe`: Only probe services that have a value of `true`
    - job_name: 'kubernetes-services'

      metrics_path: /probe
      params:
        module: [http_2xx]

      kubernetes_sd_configs:
      - api_servers:
        - 'https://kubernetes.default.svc'
        in_cluster: true
        role: service

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name

    # Example scrape config for pods
    #
    # The relabeling allows the actual pod scrape endpoint to be configured via the
    # following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
    - job_name: 'kubernetes-pods'

      kubernetes_sd_configs:
      - api_servers:
        - 'https://kubernetes.default.svc'
        in_cluster: true
        role: pod

      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_pod_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

• Logs:

time="2016-11-02T14:24:20Z" level=info msg="Starting prometheus (version=1.3.0, branch=master, revision=18254a172b1e981ed593442b2259bd63617d6aca)" source="main.go:75"
time="2016-11-02T14:24:20Z" level=info msg="Build context (go=go1.7.3, user=root@d363f050a0e0, date=20161101-17:06:27)" source="main.go:76"
time="2016-11-02T14:24:20Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:247"
time="2016-11-02T14:24:20Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in kubernetes_sd_config: in_cluster, api_servers" source="main.go:149"

[brancz]

@vorozhko thanks for reporting, we have already noticed this and will shortly adapt the config. Sorry for the inconvenience.

[jimmidyson]

@brancz I've got a config ready to submit unless you're doing it?

[brancz]

@jimmidyson go for it, I might have some discussion points, but I'm not done.

[widgetpl]

config from PR dosent work for me

time="2016-11-03T09:35:22Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): yaml: unmarshal errors:\n line 21: cannot unmarshal !!map into []*config.KubernetesSDConfig\n line 29: cannot unmarshal !!map into []*config.KubernetesSDConfig\n line 68: cannot unmarshal !!map into []*config.KubernetesSDConfig\n line 90: cannot unmarshal !!map into []*config.KubernetesSDConfig" source="main.go:149"

[jimmidyson]

Hmm those lines don't seem to relate to anything changed in the PR updated config (https://github.com/jimmidyson/prometheus/blob/da23543f29e51701b4e8ec0ffc1912e3a530c5d1/documentation/examples/prometheus-kubernetes.yml). Are the line numbers accurate in the parsing I wonder?

[widgetpl]

i have removed comments from config so it looks like

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
  namespace: infra
  labels:
    app: prometheus
    component: core
data:
  prometheus.yml: |
    scrape_configs:
    - job_name: 'kubernetes-apiservers'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
        role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)


    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
        role: endpoint

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)(?::\d+);(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name

    # Example scrape config for probing services via the Blackbox Exporter.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/probe`: Only probe services that have a value of `true`
    - job_name: 'kubernetes-services'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
        role: service

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox
      - source_labels: [__param_target]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_service_namespace]
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        target_label: kubernetes_name


    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
        role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_pod_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

sorry for that. So the lines are:

role: node
role: endpoint
role: service
role: pod

[jimmidyson]

No worries! Can you paste it into a gist to more easily view line numbers?

[widgetpl]

https://gist.github.com/widgetpl/b2c3d6dc6a3cb79d8c323365328844b0

[jimmidyson]

Looks like you've dropped the important - before role on each line the parser bailed on. I've checked the file in the PR & it is correct so must have happened when you were removing comment lines.

[widgetpl]

yes you are right probably during removing preceding - apiservers configuration.
Right now i have:

time="2016-11-03T10:08:43Z" level=error msg="Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): Unknown Kubernetes SD role \"endpoint\"" source="main.go:149"

i have updated Gist

ok i see that it is - role: endpoints

[brancz]

To follow the naming of the kubernetes API Objects it is now called endpoints rather than endpoint :)

[jimmidyson]

Yeah endpoints is correct in the PR btw.

[widgetpl]

looks like it is working now, thanks guys for helping.

[loguido]

My prometheus server is outside kubernetes cluster. With version 1.3 i cannot use apiserver option anymore. I've not understood how can i connect to apiserver...

[jimmidyson]

@loguido See the example config at https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L8. The example config wasn't updated when SD options changed unfortunately, but config in master works fine with 1.3+.

[loguido]

Thank you @jimmidyson, but as far as i can see with this config i cannot contact any kubernetes api server because my prometheus is not inside kubernetes cluster.

[fabxc]

Please check the configuration of the documentation:
https://prometheus.io/docs/operating/configuration/#<kubernetes_sd_config>
If the server_address field is left empty, we assume to be in the cluster
and will auto-detect it. If Prometheus is running outside of it, you can
simply provide it.
It should be the same configuration as before just that its now not plural
anymore and only accepts a single API server rather than a list.

On Thu, Nov 17, 2016 at 12:32 PM loguido notifications@github.com wrote:

Thank you @jimmidyson https://github.com/jimmidyson, but as far as i
can see with this config i cannot contact any kubernetes api server because
my prometheus is not inside kubernetes cluster.

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
#2147 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEuA8vRGG-Al7e_2mjYgrzEl93dCcPhVks5q_DtegaJpZM4KnTX1
.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.