[kube-prometheus-stack] cant evaluate rules , promtheus getting oom after some queries failing and very less memory usage

[navneet1075]

What did you do?

observed some weird behavior on prometheus .

What did you expect to see?

promtheus running smoothly without errors.

What did you see instead? Under which circumstances?

ts=2022-04-04T14:31:02.854Z caller=main.go:1033 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=103.243705ms db_storage=908ns remote_storage=1.381µs web_handler=474ns query_engine=797ns scrape=54.936µs scrape_sd=3.047482ms notify=15.704µs notify_sd=809.206µs rules=94.597378ms
ts=2022-04-04T14:31:03.878Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:03.883Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:03.887Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:18.819Z caller=manager.go:609 level=warn component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:34.011Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:34.113Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:34.218Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:48.631Z caller=manager.go:609 level=warn component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:03.879Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:03.883Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:03.887Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:18.632Z caller=manager.go:609 level=warn component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"

the pvc size is 50GB . and see the current data consumption of the disk when the oom happened.

du -ch *
104.0M	chunks_head
32.0K	queries.active
305.5M	wal
409.6M	total

prometheus resources

 resources:
      limits:
        cpu: "2"
        memory: 15Gi
      requests:
        cpu: "1"
        memory: 5Gi

Environment

• System information:

  Darwin 21.4.0 x86_64

• Prometheus version:

  2.31.2

• Prometheus configuration file:

- args:
    - --web.console.templates=/etc/prometheus/consoles
    - --web.console.libraries=/etc/prometheus/console_libraries
    - --storage.tsdb.retention.size=50GB
    - --config.file=/etc/prometheus/config_out/prometheus.env.yaml
    - --storage.tsdb.path=/prometheus
    - --storage.tsdb.retention.time=3h
    - --web.enable-lifecycle
    - --storage.tsdb.no-lockfile
    - --query.max-concurrency=30
    - --query.timeout=5m
    - --enable-feature=memory-snapshot-on-shutdown
    - --web.external-url=http://prometheus-operator-servic-prometheus.prometheus:9090
    - --web.route-prefix=/
    - --storage.tsdb.wal-compression

• Logs:

ts=2022-04-04T14:30:27.964Z caller=main.go:158 level=info msg="Experimental memory snapshot on shutdown enabled"
ts=2022-04-04T14:30:27.968Z caller=main.go:444 level=info msg="Starting Prometheus" version="(version=2.31.0, branch=HEAD, revision=d4c83da6d252d4edfdbd639fb817ebdb8e9ab2e4)"
ts=2022-04-04T14:30:27.968Z caller=main.go:449 level=info build_context="(go=go1.17.2, user=root@395d507f537a, date=20211102-10:28:54)"
ts=2022-04-04T14:30:27.968Z caller=main.go:450 level=info host_details="(Linux 5.10.0-9-cloud-amd64 #1 SMP Debian 5.10.83-1gardenlinux1 (2021-12-03) x86_64 prometheus-prometheus-operator-servic-prometheus-1 (none))"
ts=2022-04-04T14:30:27.968Z caller=main.go:451 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2022-04-04T14:30:27.968Z caller=main.go:452 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-04-04T14:30:27.972Z caller=web.go:542 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-04-04T14:30:27.972Z caller=main.go:839 level=info msg="Starting TSDB ..."
ts=2022-04-04T14:30:27.973Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-04-04T14:30:27.979Z caller=head.go:479 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-04-04T14:30:27.979Z caller=head.go:486 level=info component=tsdb msg="Chunk snapshot is enabled, replaying from the snapshot"
ts=2022-04-04T14:30:27.979Z caller=head.go:498 level=info component=tsdb msg="Chunk snapshot loading time" duration=23.574µs
ts=2022-04-04T14:30:28.981Z caller=head.go:513 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.002395186s
ts=2022-04-04T14:30:28.982Z caller=head.go:519 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-04-04T14:30:46.117Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=12
ts=2022-04-04T14:30:47.511Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=12
ts=2022-04-04T14:30:50.940Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=12
ts=2022-04-04T14:30:51.613Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=12
ts=2022-04-04T14:30:52.240Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=12
ts=2022-04-04T14:30:52.818Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=12
ts=2022-04-04T14:30:53.327Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=12
ts=2022-04-04T14:30:55.855Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=12
ts=2022-04-04T14:30:56.749Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=12
ts=2022-04-04T14:30:57.514Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=12
ts=2022-04-04T14:30:58.340Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=10 maxSegment=12
ts=2022-04-04T14:30:59.218Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=11 maxSegment=12
ts=2022-04-04T14:30:59.218Z caller=head.go:590 level=info component=tsdb msg="WAL segment loaded" segment=12 maxSegment=12
ts=2022-04-04T14:30:59.218Z caller=head.go:596 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=144.397µs wal_replay_duration=30.236777199s total_replay_duration=31.239413393s
ts=2022-04-04T14:31:02.661Z caller=main.go:866 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-04-04T14:31:02.661Z caller=main.go:869 level=info msg="TSDB started"
ts=2022-04-04T14:31:02.662Z caller=main.go:996 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
ts=2022-04-04T14:31:02.666Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.667Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.667Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.668Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.669Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.669Z caller=kubernetes.go:284 level=info component="discovery manager notify" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.751Z caller=main.go:1033 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=89.184161ms db_storage=1.048µs remote_storage=1.181µs web_handler=472ns query_engine=1.063µs scrape=183.746µs scrape_sd=3.460973ms notify=17.565µs notify_sd=748.299µs rules=80.482971ms
ts=2022-04-04T14:31:02.751Z caller=main.go:811 level=info msg="Server is ready to receive web requests."
ts=2022-04-04T14:31:02.751Z caller=main.go:996 level=info msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
ts=2022-04-04T14:31:02.755Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.756Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.757Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.757Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.758Z caller=kubernetes.go:284 level=info component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.758Z caller=kubernetes.go:284 level=info component="discovery manager notify" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-04-04T14:31:02.854Z caller=main.go:1033 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=103.243705ms db_storage=908ns remote_storage=1.381µs web_handler=474ns query_engine=797ns scrape=54.936µs scrape_sd=3.047482ms notify=15.704µs notify_sd=809.206µs rules=94.597378ms
ts=2022-04-04T14:31:03.878Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:03.883Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:03.887Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:18.819Z caller=manager.go:609 level=warn component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:34.011Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:34.113Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:34.218Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:31:48.631Z caller=manager.go:609 level=warn component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:03.879Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.99\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:03.883Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.9, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.9\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:03.887Z caller=manager.go:609 level=warn component="rule manager" group=kubelet.rules msg="Evaluating rule failed" rule="record: node_quantile:kubelet_pleg_relist_duration_seconds:histogram_quantile\nexpr: histogram_quantile(0.5, sum by(instance, le) (rate(kubelet_pleg_relist_duration_seconds_bucket[5m]))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"})\nlabels:\n  quantile: \"0.5\"\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"
ts=2022-04-04T14:32:18.632Z caller=manager.go:609 level=warn component="rule manager" group=kubernetes-system-kubelet msg="Evaluating rule failed" rule="alert: KubeletPodStartUpLatencyHigh\nexpr: histogram_quantile(0.99, sum by(instance, le) (rate(kubelet_pod_worker_duration_seconds_bucket{job=\"kubelet\",metrics_path=\"/metrics\"}[5m])))\n  * on(instance) group_left(node) kubelet_node_name{job=\"kubelet\",metrics_path=\"/metrics\"}\n  > 60\nfor: 15m\nlabels:\n  severity: warning\nannotations:\n  description: Kubelet Pod startup 99th percentile latency is {{ $value }} seconds\n    on node {{ $labels.node }}.\n  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeletpodstartuplatencyhigh\n  summary: Kubelet Pod startup latency is too high.\n" err="found duplicate series for the match group {instance=\"10.250.10.175:10250\"} on the right hand-side of the operation: [{__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-servic-kubelet\"}, {__name__=\"kubelet_node_name\", endpoint=\"https-metrics\", instance=\"10.250.10.175:10250\", job=\"kubelet\", metrics_path=\"/metrics\", namespace=\"kube-system\", node=\"ip-10-250-10-175.eu-west-1.compute.internal\", service=\"prometheus-operator-kube-p-kubelet\"}];many-to-many matching not allowed: matching labels must be unique on one side"

[bryanasdev000]

First, I think kube-prometheus discussion is a better place for this (https://github.com/prometheus-operator/kube-prometheus/discussions).

Second, you can query by hand and check the results and try to track down the "why", in my experience this tends to happen when there are rests of an old Prometheus install (assuming you are Working on K8S and using the above-mentioned K8S setup).

You also can disable the specified rules and check, also I don't think a rule evaluation error will trigger OOM, you can also check in Prometheus UI -> Status -> TSDB Stats to get an overview (or get a shell inside pod and run promtool tsdb analyze path/data/dir).

[roidelapluie]

It's unclear what's going on. We have fixed a bunch of on disk memory snapshots in the mean time, could be hepful if you could upgrade.

[roidelapluie]

Also, if it was crashing while querying, the queries should appear in next startup logs, which I don't see here.