Kubernetes and prometheus in different host

[prasenforu]

My prometheus server running on different host.
I was using the Link here as guide to config..
https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml

Not sure where I can give my kubernetes API servers endpoints.

in above example.

[andrewhowdencom]

You might be able to point them at the same API endpoint as the endpoint kubectl is pointing at. However, I'm not sure discovery will work as it does in the above example; You'd need to be able to route between the networks (the one k8s is using to assign ips to pods, and the one your prometheus server is running on).

This might be a good candidate for federation. https://prometheus.io/docs/operating/federation/

Suggest moving this question to StackOverflow, as it's not strictly code related, but rather docs related.

[brancz]

See the api_server field here, by default Prometheus will attempt an in cluster connection to the apiserver using the available ServiceAccount. Your instance will need to have access to the network your Pods are using to communicate (flannel, or equivalent).

[prasenforu]

Thanks @brancz
Finally I am able to setup externally Prometheus and Grafana for monitoring outside Kubernetes.

All are OK but facings problem on "kubernetes-service-endpoints", saying its down and looks like its taking internal kubernetes IP.

My prometheus config file as follows ...

--
### #prometheus.yml
# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:

  - job_name: 'Kubenetes-Hosts'
    static_configs:
      - targets: ['ip-10-52-2-56.ap-northeast-2.compute.internal:9100','ip-10-52-2-59.ap-northeast-2.compute.internal:9100','ip-10-52-2-54.ap-northeast-2.com
pute.internal:9100']
    relabel_configs:
       - source_labels: [ __address__ ]
         target_label: instance
         regex: ip-10-52-2-56.ap-northeast-2.compute.internal:9100
         replacement: Master
       - source_labels: [ __address__ ]
         target_label: instance
         regex: ip-10-52-2-59.ap-northeast-2.compute.internal:9100
         replacement: Node-1
       - source_labels: [ __address__ ]
         target_label: instance
         regex: ip-10-52-2-54.ap-northeast-2.compute.internal:9100
         replacement: Node-2

  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
    - api_server: ip-10-52-2-56.ap-northeast-2.compute.internal:8080
      role: node
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - source_labels: [__address__]
      regex: '(.*):10250'
      replacement: '${1}:10255'
      target_label: __address__
    metric_relabel_configs:
    - source_labels: [io_kubernetes_container_name,container_name]
      action: replace
      regex: (.*);(.*)
      replacement: '${1}${2}'
      target_label: io_kubernetes_container_name
    - source_labels: [kubernetes_pod_name,pod_name]
      action: replace
      regex: (.*);(.*)
      replacement: '${1}${2}'
      target_label: kubernetes_pod_name
    - source_labels: [kubernetes_pod_name]
      action: replace
      target_label: io_kubernetes_pod_name

  - job_name: 'kubernetes-node-exporter'
    kubernetes_sd_configs:
    - api_server: ip-10-52-2-56.ap-northeast-2.compute.internal:8080
      role: node
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - source_labels: [__meta_kubernetes_role]
      action: replace
      target_label: kubernetes_role
    - source_labels: [__address__]
      regex: '(.*):10250'
      replacement: '${1}:9100'
      target_label: __address__
    - source_labels: [__meta_kubernetes_node_label_kubernetes_io_hostname]
      target_label: __instance__
      # set "name" value to "job"
    - source_labels: [job]
      regex: 'kubernetes-(.*)'
      replacement: '${1}'
      target_label: name

  - job_name: 'kubernetes-service-endpoints'
    kubernetes_sd_configs:
    - api_server: ip-10-52-2-56.ap-northeast-2.compute.internal:8080
      role: endpoints
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
      action: replace
      target_label: __scheme__
      regex: (https?)
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: (.+)(?::\d+);(\d+)
      replacement: $1:$2
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_service_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: kubernetes_name

  - job_name: 'kubernetes-services'
    metrics_path: /probe
    params:
      module: [http_2xx]
    kubernetes_sd_configs:
    - api_server: ip-10-52-2-56.ap-northeast-2.compute.internal:8080
      role: service
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
      action: keep
      regex: true
    - source_labels: [__address__]
      target_label: __param_target
    - target_label: __address__
      replacement: blackbox
    - source_labels: [__param_target]
      target_label: instance
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_service_namespace]
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_service_name]
      target_label: kubernetes_name

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - api_server: ip-10-52-2-56.ap-northeast-2.compute.internal:8080
      role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: (.+):(?:\d+);(\d+)
      replacement: ${1}:${2}
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_pod_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

--

[brancz]

Your Prometheus instance needs access to the private network your Kubernetes cluster is using, for example if you are using flannel, then you need to add the machine Prometheus is running on to that flannel network. Once you can ping those private IPs from that machine this discovery should also work.

[prasenforu]

That mean one pod/container (Prometheus) need to run inside kuberetes cluster ?
If its yes!!
Then we have two Prometheus servers/pod. one is inside of kubernetes cluster and another is outside of cluster.

[brancz]

That would be a possibility having Prometheus running inside the cluster is certainly the most common practice when you monitor Kubernetes and things running on top of Kubernetes, but it would also work if your machine outside of the Kubernetes cluster is just part of the network to be able to route those IPs.

[prasenforu]

why keeping outside bcoz we are not monitoring only kubernetes also other things also monitoring.

if I run any kubernetes pod/container in side cluster what will be config file inside prometheus, only
target: localhost:9090 will work ?

Any configuration need to change in external prometheus ?

or any alternate approch ?

[brancz]

You can still monitor things outside of Kubernetes when using a Prometheus that is running inside of Kubernetes. Otherwise there is nothing wrong with running two instances of Prometheus, one for monitoring targets inside of Kubernetes and one for monitoring targets outside of Prometheus.

if I run any kubernetes pod/container in side cluster what will be config file inside prometheus, only
target: localhost:9090 will work ?

I don't understand this, could you please rephrase it?

[prasenforu]

There was a problem in kubedns thats why endpoint fails. After dns issue resolved all ok Thanks & regards, Prasenjit On 3 Mar 2017, at 7:12 p.m., Frederic Branczyk <notifications@github.com<mailto:notifications@github.com>> wrote: You can still monitor things outside of Kubernetes when using a Prometheus that is running inside of Kubernetes. Otherwise there is nothing wrong with running two instances of Prometheus, one for monitoring targets inside of Kubernetes and one for monitoring targets outside of Prometheus. if I run any kubernetes pod/container in side cluster what will be config file inside prometheus, only target: localhost:9090 will work ? I don't understand this, could you please rephrase it? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2430 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASbWgfReJm2KFi_QUpgHbctMREqKRb-Zks5riBjMgaJpZM4MBrA9>.

[brancz]

Great! Can we close this issue here then?

[prasenforu]

Yes please. Thanks & regards, Prasenjit On 7 Mar 2017, at 1:56 p.m., Frederic Branczyk <notifications@github.com<mailto:notifications@github.com>> wrote: Great! Can we close this issue here then? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2430 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASbWgW3McXHNL-wBFxZMgN_0Y9nBUinLks5rjRS_gaJpZM4MBrA9>.

[hiscal2015]

@prasenforu I also need to run Prometheus outside of Kubernetes cluster, but how to deal with the token? I see you only defined api server address in the config file.

[prasenforu]

Suggest allways use inside kubernetes.
I was using external without security (non ssl) for that no token required.

[greenled]

Also having troubles with token. @prasenforu did you find anything about it?

[costimuraru]

We also have the Prometheus server outside of k8s. Any idea how to deal with the token and make it work?

[lnformer]

i think the documentation is laking of how to properly monitor externally kubernetes with prometheus.

[damien-roche]

Utterly confused this hasn't been addressed yet. Everybody is monitoring Kube from inside Kube?

I have multiple Kube clusters and I have random nodes dotted around (apps, postgres, rabbitmq, etc). I have a central Prometheus server which pulls in metrics from my nodes no problem. I don't want another Prometheus instance chewing up ram on every Kube cluster; I already have an instance.

"Otherwise there is nothing wrong with running two instances of Prometheus, one for monitoring targets inside of Kubernetes and one for monitoring targets outside of Prometheus."

So if we have multiple Kube clusters we will have a Prometheus instance for each cluster? What if you have 5 clusters? Now I have to manage 5 different Prometheus instances each related to a different cluster? I just want centralised monitoring.

Can somebody in the know please document this. I don't understand how it isn't a common use-case.

EDIT. I have turned up some possible solutions.

An answer on SO suggests a federation setup. Prometheus inside your cluster, expose metrics from that to outside the cluster and your central/external Prometheus instance(s). (https://stackoverflow.com/a/47643005/419017)

There is also a project here which exposes cluster level metrics: https://github.com/kubernetes/kube-state-metrics

Hope that gives someone else something to run with.

[prasenforu]

But anyway we have to run Prometheus in each cluster. That I do not want . Basically I want to run Prometheus completely out of kubernetes cluster.