[complete stack] Stack is not working by default on K3S clusters, enable it to work out of the box on K3S

[borgdrone7]

Is your feature request related to a problem ?

Hi, I am trying to setup kube prometheus stack on K3s and I am having a lot of issues. By default kube controller, proxy and etcd services are not mapped to any endpoint (with the latest version of the stack) and I have empty dashboards for those services in Grafana. Also I read that K3s is exposing metrics from all these in each of them so for the ones which work I will get duplicated data in the Prometheus. So is there custom version of kube prometheus stack which works with K3s and if not, are there any workarounds to get it to work with the latest prom stack version?

Describe the solution you'd like.

I don't have a solution. I noticed that if I add node IP for the services which do not work (proxy, controller, etcd) to endpoints key in values yaml I get Grafana dashboard showing values. I only tested it with version 41 of the stack. But this solution is not longterm and doesn't seem like something I could count on. Didn't test with multi node setup and not sure how would that work. Also I don't understand why it even works (I guess because k3s exposes all variables on all services but not sure)

Describe alternatives you've considered.

I don't know of any alternatives.

Additional context.

No response

[Camphul]

This helm chart pulls metrics from services not related to prometheus. To configure k3s with this chart you will have to add certain arguments to the k3s service (see k3s docs, expose etcd metrics to 0.0.0.0 as an example). Then configure this chart's applicable endpoints to your node ip's.

Issue can be closed?

[borgdrone7]

This helm chart pulls metrics from services not related to prometheus. To configure k3s with this chart you will have to add certain arguments to the k3s service (see k3s docs, expose etcd metrics to 0.0.0.0 as an example). Then configure this chart's applicable endpoints to your node ip's.

Issue can be closed?

Hi, I exposed all the k3s metrics by configuring it, however duplicated data is still there, so if anyone is using this chart by just simply configuring k3s and the endpoints then he will get a lot of wasted resources. I partially solved it by adding some drop filters in chart's Prometheus scraper configuration. The point is that the stack is not working out of the box with k3s and it is not easy to make it work correctly (not that I say it should, but it would be nice if it did). I see on the net that people mostly ignore this duplication which is sub optimal as they will get 4x the data in their system, or they do things like disabling all metrics in k3s except one in which case they loose ability to easily upgrade the stack as they need to manually configure Prometheus scraping and Grafana dashboards, basically completely changing the stack. I asked k3s developers and they told me that I should enable only one metrics endpoint to avoid duplicates, which in turn beats the purpose of this stack as too much of the stuff needs to be changed.
Regarding endpoints I did add all nodes IPs to the chart's endpoints configuration, however this requires that metrics are exposed on 0.0.0.0 which might be a security issue (not 100% sure but it looks like it as Prometheus by default has no authentication of any kind). Do you have any idea how to enable stack to scrape the data without exposing the metrics on 0.0.0.0? Or is there some easy way to expose them and still secure these endpoints?
Thanks

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[onedr0p]

@borgdrone7 are you willing to share your helm values for dropping the duplicate metrics? It would help out greatly. Thanks!

[borgdrone7]

@borgdrone7 are you willing to share your helm values for dropping the duplicate metrics? It would help out greatly. Thanks!

Hi, for metrics service instead of just keeping the metrics which are prefixed with the service name, I actually dropped metrics from other services. That's because if we only keep metrics with service prefix, then some common values used in Grafana will also be dropped so the charts will be incomplete. This could be done better by figuring out exactly which metrics we want to keep, but I checked rows added and this method drops significant amount of duplicated stuff and all charts still work.
So for each service just drop any other service prefix. Here is example for kubeProxy and kubeEtcd, you can do accordingly with others (apiserver, kubelet, scheduler).

kubeProxy:
    metricRelabelings:
      - action: drop
        regex: '(apiserver_.*|kubelet_.*|scheduler_.*|etcd_.*)'
        sourceLabels: [__name__]

kubeEtcd:
    metricRelabelings:
      - action: drop
        regex: '(apiserver_.*|kubelet_.*|kubeproxy_.*|scheduler_.*)'
        sourceLabels: [__name__]

For controller manager I used:

kubeControllerManager:
    metricRelabelings:
      - action: drop
        regex: '(apiserver_.*|kubelet_.*|scheduler_.*|kubeproxy_.*|etcd_.*)'
        sourceLabels: [__name__]

Hope this helps.

[onedr0p]

I was able to do some work in trying to reduce the duplicated metrics. This has reduced prometheus memory usage from around 6-7GB to around 2-3GB.

Here is what I use currently:

    kubelet:
      enabled: true
      serviceMonitor:
        metricRelabelings:
          # Remove duplicate metrics
          - sourceLabels: ["__name__"]
            regex: "(apiserver_audit|apiserver_client|apiserver_delegated|apiserver_envelope|apiserver_storage|apiserver_webhooks|authentication_token|cadvisor_version|container_blkio|container_cpu|container_fs|container_last|container_memory|container_network|container_oom|container_processes|container|csi_operations|disabled_metric|get_token|go|hidden_metric|kubelet_certificate|kubelet_cgroup|kubelet_container|kubelet_containers|kubelet_cpu|kubelet_device|kubelet_graceful|kubelet_http|kubelet_lifecycle|kubelet_managed|kubelet_node|kubelet_pleg|kubelet_pod|kubelet_run|kubelet_running|kubelet_runtime|kubelet_server|kubelet_started|kubelet_volume|kubernetes_build|kubernetes_feature|machine_cpu|machine_memory|machine_nvm|machine_scrape|node_namespace|plugin_manager|prober_probe|process_cpu|process_max|process_open|process_resident|process_start|process_virtual|registered_metric|rest_client|scrape_duration|scrape_samples|scrape_series|storage_operation|volume_manager|volume_operation|workqueue)_(.+)"
            action: keep
          - sourceLabels: ["node"]
            targetLabel: instance
            action: replace
    kubeApiServer:
      enabled: true
      serviceMonitor:
        metricRelabelings:
          # Remove duplicate metrics
          - sourceLabels: ["__name__"]
            regex: "(aggregator_openapi|aggregator_unavailable|apiextensions_openapi|apiserver_admission|apiserver_audit|apiserver_cache|apiserver_cel|apiserver_client|apiserver_crd|apiserver_current|apiserver_envelope|apiserver_flowcontrol|apiserver_init|apiserver_kube|apiserver_longrunning|apiserver_request|apiserver_requested|apiserver_response|apiserver_selfrequest|apiserver_storage|apiserver_terminated|apiserver_tls|apiserver_watch|apiserver_webhooks|authenticated_user|authentication|disabled_metric|etcd_bookmark|etcd_lease|etcd_request|field_validation|get_token|go|grpc_client|hidden_metric|kube_apiserver|kubernetes_build|kubernetes_feature|node_authorizer|pod_security|process_cpu|process_max|process_open|process_resident|process_start|process_virtual|registered_metric|rest_client|scrape_duration|scrape_samples|scrape_series|serviceaccount_legacy|serviceaccount_stale|serviceaccount_valid|watch_cache|workqueue)_(.+)"
            action: keep
          # Remove high cardinality metrics
          - sourceLabels: ["__name__"]
            regex: (apiserver|etcd|rest_client)_request(|_sli|_slo)_duration_seconds_bucket
            action: drop
          - sourceLabels: ["__name__"]
            regex: (apiserver_response_sizes_bucket|apiserver_watch_events_sizes_bucket)
            action: drop
    kubeControllerManager:
      enabled: true
      endpoints: ["192.168.42.10", "192.168.42.11", "192.168.42.12"]
      serviceMonitor:
        metricRelabelings:
          # Remove duplicate metrics
          - sourceLabels: ["__name__"]
            regex: "(apiserver_audit|apiserver_client|apiserver_delegated|apiserver_envelope|apiserver_storage|apiserver_webhooks|attachdetach_controller|authenticated_user|authentication|cronjob_controller|disabled_metric|endpoint_slice|ephemeral_volume|garbagecollector_controller|get_token|go|hidden_metric|job_controller|kubernetes_build|kubernetes_feature|leader_election|node_collector|node_ipam|process_cpu|process_max|process_open|process_resident|process_start|process_virtual|pv_collector|registered_metric|replicaset_controller|rest_client|retroactive_storageclass|root_ca|running_managed|scrape_duration|scrape_samples|scrape_series|service_controller|storage_count|storage_operation|ttl_after|volume_operation|workqueue)_(.+)"
            action: keep
    kubeEtcd:
      enabled: true
      endpoints: ["192.168.42.10", "192.168.42.11", "192.168.42.12"]
    kubeProxy:
      enabled: false # Disabled because eBPF
      endpoints: ["192.168.42.10", "192.168.42.11", "192.168.42.12"]
    kubeScheduler:
      enabled: true
      endpoints: ["192.168.42.10", "192.168.42.11", "192.168.42.12"]
      serviceMonitor:
        metricRelabelings:
          # Remove duplicate metrics
          - sourceLabels: ["__name__"]
            regex: "(apiserver_audit|apiserver_client|apiserver_delegated|apiserver_envelope|apiserver_storage|apiserver_webhooks|authenticated_user|authentication|disabled_metric|go|hidden_metric|kubernetes_build|kubernetes_feature|leader_election|process_cpu|process_max|process_open|process_resident|process_start|process_virtual|registered_metric|rest_client|scheduler|scrape_duration|scrape_samples|scrape_series|workqueue)_(.+)"
            action: keep

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.

[maradanasai]

Thanks @onedr0p for this. Anything latest on this? also, can you please share more details on why kubeProxy is disabled due to eBPF ?