Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[mimir-distributed] Frequently Gaps Inside Mimir Metrics #1517

Open
ghost opened this issue Jun 20, 2022 · 9 comments
Open

[mimir-distributed] Frequently Gaps Inside Mimir Metrics #1517

ghost opened this issue Jun 20, 2022 · 9 comments

Comments

@ghost
Copy link

ghost commented Jun 20, 2022 •
edited by ghost
Loading

Hello,

we got an running instance of mimir-distributed inside our kubernetes cluster. But there are still some problems. It seems to me that the samples, which are out of order, can not get ingested. I think prometheus tries to remote write old samples from its WAL. What do i need to configure in mimir to allow prometheus to send these old samples? How do i archive a gapless traffic of samples? If i take a look at mimir trough grafana i got some empty gaps (time ranges with no data) in the graph for every metric. What am i doing wrong here?

I added our config. Also logs of the mimir-distributed-distributor, mimir-distributed-ingester and prometheus-rancher-monitoring-prometheus pods.

Best Regards
@ghost
Copy link
Author

ghost commented Jun 20, 2022

Active Alerts:

  • warning | MimirRequestLatency | mimir-distributed-distributor-headless api_v1_push is experiencing 9.88s 99th percentile latency.
  • warning | MimirRequestLatency | mimir-distributed-gossip-ring api_v1_push is experiencing 9.83s 99th percentile latency.
  • warning | MimirRequestLatency | mimir-distributed-distributor api_v1_push is experiencing 9.86s 99th percentile latency.
  • warning | MimirStoreGatewayNoSyncedTenants | Mimir store-gateway mimir-distributed-store-gateway-0 in /cattle-monitoring-system is not syncing any blocks for any tenant.
  • warning | PrometheusRemoteWriteDesiredShards

@ghost
Copy link
Author

ghost commented Jun 20, 2022

Config:

mimir:
  config: >
    {{- if not .Values.enterprise.enabled -}}

    multitenancy_enabled: false

    {{- end }}


    limits:
      compactor_blocks_retention_period: 604800s
      ingestion_rate: 60000
      max_global_series_per_metric: 2000000
      max_global_series_per_user: 2000000
      max_label_names_per_series: 60

    distributor:
      instance_limits:
        max_ingestion_rate: 0
        max_inflight_push_requests: 0
      remote_timeout: 30s

    ...

@ghost
Copy link
Author

ghost commented Jun 20, 2022

mimir-distributed-distributor log:

Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227192795Z caller=logging.go:72 traceID=1736da2b0734600d msg="POST /api/v1/push (500) 1.722399471s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15250; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227707732Z caller=logging.go:72 traceID=6e63569a6618e2db msg="POST /api/v1/push (500) 1.913782544s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16030; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227194307Z caller=logging.go:72 traceID=06c2a40f7d29110b msg="POST /api/v1/push (500) 486.407113ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14918; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227734096Z caller=logging.go:72 traceID=4576d9fdd7287b4b msg="POST /api/v1/push (500) 1.119823902s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16185; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227736782Z caller=logging.go:72 traceID=47da02c9af0d1a91 msg="POST /api/v1/push (500) 1.019907518s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14810; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227791683Z caller=logging.go:72 traceID=00807a367d15eb3f msg="POST /api/v1/push (500) 416.256225ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13813; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227814911Z caller=logging.go:72 traceID=1fa857fb9629012c msg="POST /api/v1/push (500) 510.711781ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14655; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227424118Z caller=logging.go:72 traceID=0bfd3f1a63b22e98 msg="POST /api/v1/push (500) 2.01935168s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15572; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22782542Z caller=logging.go:72 traceID=0dfdd1bf9c261c4f msg="POST /api/v1/push (500) 2.104112757s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15516; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227498668Z caller=logging.go:72 traceID=27a84d3a1f41e2de msg="POST /api/v1/push (500) 2.216045364s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15175; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227448365Z caller=logging.go:72 traceID=5143feb6c199d63a msg="POST /api/v1/push (500) 1.592792258s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13028; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227747108Z caller=logging.go:72 traceID=47ab797531315361 msg="POST /api/v1/push (500) 2.314307316s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13239; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227876713Z caller=logging.go:72 traceID=099b9513c7b0d731 msg="POST /api/v1/push (500) 318.832623ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14863; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227768142Z caller=logging.go:72 traceID=74c404f3a2e04a89 msg="POST /api/v1/push (500) 1.312822963s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15778; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227760562Z caller=logging.go:72 traceID=1816442083514f24 msg="POST /api/v1/push (500) 517.110517ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12800; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227770761Z caller=logging.go:72 traceID=1c04b551a4d83ce8 msg="POST /api/v1/push (500) 122.295527ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13004; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22777875Z caller=logging.go:72 traceID=72cfe829ef2a0e6f msg="POST /api/v1/push (500) 1.212204093s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12082; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.227472749Z caller=logging.go:72 traceID=081936b6df936bab msg="POST /api/v1/push (500) 696.227745ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14713; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228180425Z caller=logging.go:72 traceID=5518a048bdaa5c5b msg="POST /api/v1/push (500) 1.690873069s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13039; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228243271Z caller=logging.go:72 traceID=7c46d278cadc9c00 msg="POST /api/v1/push (500) 1.613758561s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 13927; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228248336Z caller=logging.go:72 traceID=3a18e7eb8f43f969 msg="POST /api/v1/push (500) 3.091865069s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16164; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228419214Z caller=logging.go:72 traceID=29fc982db856b100 msg="POST /api/v1/push (500) 1.017792748s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12572; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228430603Z caller=logging.go:72 traceID=2673eab65d91e83b msg="POST /api/v1/push (500) 1.121916731s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15985; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228448025Z caller=logging.go:72 traceID=72e582882f8b065e msg="POST /api/v1/push (500) 2.722082583s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 17358; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228496778Z caller=logging.go:72 traceID=1c52bcdb21cf5efa msg="POST /api/v1/push (500) 1.909129961s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12981; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228551919Z caller=logging.go:72 traceID=6c6ced0953f83e79 msg="POST /api/v1/push (500) 2.020535714s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 11016; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228598416Z caller=logging.go:72 traceID=757194e641fa476e msg="POST /api/v1/push (500) 3.321319526s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16142; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22871595Z caller=logging.go:72 traceID=4757ec01b60357c4 msg="POST /api/v1/push (500) 2.516569804s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14389; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228791793Z caller=logging.go:72 traceID=701dd0f19ce9f80d msg="POST /api/v1/push (500) 86.595219ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16778; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.228874816Z caller=logging.go:72 traceID=428fc95d7b0bb204 msg="POST /api/v1/push (500) 2.514256872s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14006; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229037391Z caller=logging.go:72 traceID=469e54e69e538a94 msg="POST /api/v1/push (500) 520.816824ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14051; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229044499Z caller=logging.go:72 traceID=1b7dbca99112c312 msg="POST /api/v1/push (500) 518.300411ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 12953; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229054522Z caller=logging.go:72 traceID=6eec49cd41925e6a msg="POST /api/v1/push (500) 613.843239ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14773; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.22923861Z caller=logging.go:72 traceID=1857fecda2a465c9 msg="POST /api/v1/push (500) 3.419017489s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14995; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229349614Z caller=logging.go:72 traceID=21b65e111d5cfb75 msg="POST /api/v1/push (500) 3.015330856s Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14005; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.229356677Z caller=logging.go:72 traceID=6e33820aa370fa2f msg="POST /api/v1/push (500) 720.936237ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 14085; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.233139646Z caller=logging.go:72 traceID=6dc90806423655a2 msg="POST /api/v1/push (500) 19.089005ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 15493; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "
Mon, 20 Jun 2022 12:31:51 | level=warn ts=2022-06-20T10:31:51.241303811Z caller=logging.go:72 traceID=497a23a70122d437 msg="POST /api/v1/push (500) 27.302655ms Response: \"context canceled\\n\" ws: false; Connection: close; Content-Encoding: snappy; Content-Length: 16582; Content-Type: application/x-protobuf; User-Agent: Prometheus/2.28.1; X-Prometheus-Remote-Write-Version: 0.1.0; X-Scope-Orgid: 0; "

@ghost
Copy link
Author

ghost commented Jun 20, 2022

mimir-distributed-ingester log:

Mon, 20 Jun 2022 13:32:39 | level=warn ts=2022-06-20T11:32:39.189854883Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=726.413µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:07:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"rbac.authorization.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.9\", resource=\"roles\", scope=\"namespace\", service=\"kubernetes\", verb=\"POST\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:33:54 | level=warn ts=2022-06-20T11:33:54.711395544Z caller=grpc_logging.go:38 duration=696.55µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"apiregistration.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"apiservices\", scope=\"cluster\", service=\"kubernetes\", verb=\"DELETE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:33:58 | level=warn ts=2022-06-20T11:33:58.51486206Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=670.173µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"monitoring.coreos.com\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"alertmanagers\", scope=\"namespace\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:12 | level=warn ts=2022-06-20T11:34:12.589178909Z caller=grpc_logging.go:38 duration=510.352µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"admissionregistration.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"mutatingwebhookconfigurations\", scope=\"cluster\", service=\"kubernetes\", verb=\"GET\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:15 | level=warn ts=2022-06-20T11:34:15.551402567Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=748.994µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"catalog.cattle.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"operations\", scope=\"namespace\", service=\"kubernetes\", verb=\"DELETE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:15 | level=warn ts=2022-06-20T11:34:15.831319781Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=805.451µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:32.121Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"web-app\", destination_canonical_revision=\"1938\", destination_canonical_service=\"web-app\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"web-app.testing.svc.cluster.local\", destination_service_name=\"web-app\", destination_service_namespace=\"testing\", destination_version=\"1938\", destination_workload=\"web-app\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", le=\"600000\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:20 | level=warn ts=2022-06-20T11:34:20.166798909Z caller=grpc_logging.go:38 duration=615.585µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:13.201Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-default\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-default\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.14.43:15090\", job=\"elasticsearch-es-transport\", le=\"2500\", namespace=\"testing\", pod=\"elasticsearch-es-default-0\", pod_name=\"elasticsearch-es-default-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-transport\", source_app=\"flow-executor\", source_canonical_revision=\"1974\", source_canonical_service=\"flow-executor\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"1974\", source_workload=\"flow-executor\", source_workload_namespace=\"testing\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:23 | level=warn ts=2022-06-20T11:34:23.211139804Z caller=grpc_logging.go:38 duration=521.226µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:23.611Z, series={__name__=\"istio_response_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-default\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-default\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.9.196:15090\", job=\"elasticsearch-es-transport\", le=\"300000\", namespace=\"testing\", pod=\"elasticsearch-es-default-1\", pod_name=\"elasticsearch-es-default-1\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-transport\", source_app=\"flow-executor\", source_canonical_revision=\"1945\", source_canonical_service=\"flow-executor\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"1945\", source_workload=\"flow-executor\", source_workload_namespace=\"testing\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:25 | level=warn ts=2022-06-20T11:34:25.259276937Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=439.795µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:20.335Z, series={__name__=\"envoy_server_live\", container=\"istio-proxy\", endpoint=\"15090\", instance=\"10.42.9.41:15090\", job=\"audit\", namespace=\"testing\", pod=\"audit-85d7fb579d-gk588\", pod_name=\"audit-85d7fb579d-gk588\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", service=\"audit\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:26 | level=warn ts=2022-06-20T11:34:26.288849589Z caller=grpc_logging.go:38 duration=514.87µs method=/cortex.Ingester/Push err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:47.121Z, series={__name__=\"istio_request_duration_milliseconds_bucket\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"flow\", destination_canonical_revision=\"1911\", destination_canonical_service=\"flow\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"flow.testing.svc.cluster.local\", destination_service_name=\"flow\", destination_service_namespace=\"testing\", destination_version=\"1911\", destination_workload=\"flow\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", le=\"0.5\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"304\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:34:41 | level=warn ts=2022-06-20T11:34:41.442058796Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=573.971µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"pods\", verb=\"read\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:22 | level=warn ts=2022-06-20T11:35:22.792209788Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=495.438µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"policy\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"podsecuritypolicies\", scope=\"cluster\", service=\"kubernetes\", verb=\"POST\", version=\"v1beta1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:39 | level=warn ts=2022-06-20T11:35:39.879543641Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=953.927µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"storage.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"volumeattachments\", scope=\"cluster\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:46 | level=warn ts=2022-06-20T11:35:46.674285571Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=491.576µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"networking.istio.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.9\", resource=\"virtualservices\", scope=\"namespace\", service=\"kubernetes\", verb=\"GET\", version=\"v1beta1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:48 | level=warn ts=2022-06-20T11:35:48.242767891Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=577.645µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:49.922Z, series={__name__=\"envoy_listener_manager_lds_update_duration_bucket\", container=\"istio-proxy\", endpoint=\"15090\", instance=\"10.42.9.231:15090\", job=\"secret\", le=\"250\", namespace=\"cloud\", pod=\"secret-6b844b8985-p52zg\", pod_name=\"secret-6b844b8985-p52zg\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", service=\"secret\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:55 | level=warn ts=2022-06-20T11:35:55.173285815Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=691.691µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"apps\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"replicasets\", scope=\"namespace\", service=\"kubernetes\", verb=\"PUT\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:35:57 | level=warn ts=2022-06-20T11:35:57.575252365Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=915.975µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.9\", resource=\"services\", scope=\"namespace\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:01 | level=warn ts=2022-06-20T11:36:01.000311337Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=720.084µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"rbac.authorization.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.5\", resource=\"clusterroles\", scope=\"cluster\", service=\"kubernetes\", verb=\"DELETE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:08 | level=warn ts=2022-06-20T11:36:08.380644566Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=596.909µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:47.121Z, series={__name__=\"istio_request_duration_milliseconds_count\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"timeseries\", destination_canonical_revision=\"1973\", destination_canonical_service=\"timeseries\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"timeseries.testing.svc.cluster.local\", destination_service_name=\"timeseries\", destination_service_namespace=\"testing\", destination_version=\"1973\", destination_workload=\"timeseries\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"403\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:15 | level=warn ts=2022-06-20T11:36:15.88950191Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=520.855µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:08:48.703Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-transform\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/sealed-services/sa/default\", destination_service=\"elasticsearch-es-http.sealed-services.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"sealed-services\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-transform\", destination_workload_namespace=\"sealed-services\", endpoint=\"15090\", instance=\"10.42.12.169:15090\", job=\"elasticsearch-es-transform\", le=\"50\", namespace=\"sealed-services\", pod=\"elasticsearch-es-transform-0\", pod_name=\"elasticsearch-es-transform-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-transform\", source_app=\"unknown\", source_canonical_revision=\"latest\", source_canonical_service=\"elasticsearch-kb\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/sealed-services/sa/default\", source_version=\"unknown\", source_workload=\"elasticsearch-kb\", source_workload_namespace=\"sealed-services\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:25 | level=warn ts=2022-06-20T11:36:25.27399867Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=469.65µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:47.121Z, series={__name__=\"istio_response_bytes_bucket\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"asset\", destination_canonical_revision=\"1958\", destination_canonical_service=\"asset\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"asset.testing.svc.cluster.local\", destination_service_name=\"asset\", destination_service_namespace=\"testing\", destination_version=\"1958\", destination_workload=\"asset\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", le=\"10000\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:33 | level=warn ts=2022-06-20T11:36:33.279107798Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=650.877µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:45.424Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-transform\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-transform\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.94:15090\", job=\"elasticsearch-es-http\", le=\"5\", namespace=\"testing\", pod=\"elasticsearch-es-transform-0\", pod_name=\"elasticsearch-es-transform-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"409\", response_flags=\"-\", service=\"elasticsearch-es-http\", source_app=\"unknown\", source_canonical_revision=\"latest\", source_canonical_service=\"elasticsearch-kb\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"unknown\", source_workload=\"elasticsearch-kb\", source_workload_namespace=\"testing\"}" msg=gRPC
Mon, 20 Jun 2022 13:36:39 | level=warn ts=2022-06-20T11:36:39.820365459Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=593.632µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"validatingwebhookconfigurations\", verb=\"read\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:37 | level=warn ts=2022-06-20T11:37:37.549507926Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=568.603µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:11:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"kibana.k8s.elastic.co\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"kibanas\", scope=\"namespace\", service=\"kubernetes\", verb=\"PATCH\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:47 | level=warn ts=2022-06-20T11:37:47.819143996Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=477.755µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"rbac.authorization.k8s.io\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"clusterrolebindings\", scope=\"cluster\", service=\"kubernetes\", verb=\"GET\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:48 | level=warn ts=2022-06-20T11:37:48.48279175Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=762.907µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:10:15.362Z, series={__name__=\"cluster_quantile:apiserver_request_duration_seconds:histogram_quantile\", component=\"apiserver\", endpoint=\"https\", group=\"kibana.k8s.elastic.co\", job=\"apiserver\", namespace=\"default\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", quantile=\"0.99\", resource=\"kibanas\", scope=\"namespace\", service=\"kubernetes\", verb=\"UPDATE\", version=\"v1\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:51 | level=warn ts=2022-06-20T11:37:51.727862428Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=610.987µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:09:47.121Z, series={__name__=\"istio_requests_total\", connection_security_policy=\"unknown\", container=\"istio-proxy\", destination_app=\"alert\", destination_canonical_revision=\"1963\", destination_canonical_service=\"alert\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"alert.testing.svc.cluster.local\", destination_service_name=\"alert\", destination_service_namespace=\"testing\", destination_version=\"1963\", destination_workload=\"alert\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.194:15090\", job=\"istio-ingressgateway\", namespace=\"istio-system\", pod=\"istio-ingressgateway-c594d897-g2h7q\", pod_name=\"istio-ingressgateway-c594d897-g2h7q\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"source\", request_protocol=\"http\", response_code=\"403\", response_flags=\"-\", service=\"istio-ingressgateway\", source_app=\"istio-ingressgateway\", source_canonical_revision=\"latest\", source_canonical_service=\"istio-ingressgateway\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account\", source_version=\"unknown\", source_workload=\"istio-ingressgateway\", source_workload_namespace=\"istio-system\"}" msg=gRPC
Mon, 20 Jun 2022 13:37:53 | level=warn ts=2022-06-20T11:37:53.276950969Z caller=grpc_logging.go:38 method=/cortex.Ingester/Push duration=586.305µs err="rpc error: code = Code(400) desc = user=anonymous: err: out of order sample. timestamp=2022-06-20T11:11:27.026Z, series={__name__=\"istio_request_bytes_bucket\", connection_security_policy=\"mutual_tls\", container=\"istio-proxy\", destination_app=\"unknown\", destination_canonical_revision=\"latest\", destination_canonical_service=\"elasticsearch-es-transform\", destination_cluster=\"Kubernetes\", destination_principal=\"spiffe://cluster.local/ns/testing/sa/default\", destination_service=\"elasticsearch-es-http.testing.svc.cluster.local\", destination_service_name=\"elasticsearch-es-http\", destination_service_namespace=\"testing\", destination_version=\"unknown\", destination_workload=\"elasticsearch-es-transform\", destination_workload_namespace=\"testing\", endpoint=\"15090\", instance=\"10.42.12.94:15090\", job=\"elasticsearch-es-internal-http\", le=\"2500\", namespace=\"testing\", pod=\"elasticsearch-es-transform-0\", pod_name=\"elasticsearch-es-transform-0\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\", reporter=\"destination\", request_protocol=\"http\", response_code=\"200\", response_flags=\"-\", service=\"elasticsearch-es-internal-http\", source_app=\"asset\", source_canonical_revision=\"1945\", source_canonical_service=\"asset\", source_cluster=\"Kubernetes\", source_principal=\"spiffe://cluster.local/ns/testing/sa/default\", source_version=\"1945\", source_workload=\"asset\", source_workload_namespace=\"testing\"}" msg=gRPC

@ghost
Copy link
Author

ghost commented Jun 20, 2022

prometheus-rancher-monitoring-prometheus log:

level=info ts=2022-06-20T11:38:33.057Z caller=main.go:443 msg="Starting Prometheus" version="(version=2.28.1, branch=HEAD, revision=b0944590a1c9a6b35dc5a696869f75f422b107a1)"
--
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:448 build_context="(go=go1.16.5, user=root@2915dd495090, date=20210701-15:20:10)"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:449 host_details="(Linux 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 prometheus-rancher-monitoring-prometheus-0 (none))"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:450 fd_limits="(soft=1048576, hard=1048576)"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.057Z caller=main.go:451 vm_limits="(soft=unlimited, hard=unlimited)"
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.061Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.061Z caller=main.go:824 msg="Starting TSDB ..."
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.062Z caller=tls_config.go:227 component=web msg="TLS is disabled." http2=false
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.062Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655056800062 maxt=1655121600000 ulid=01G5F0837YAC495GQ50V4FTZTV
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.063Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655121600063 maxt=1655186400000 ulid=01G5GY1QX1Z92BN4GPHK997M1A
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.063Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655186400014 maxt=1655251200000 ulid=01G5JVVEYCDFFMVEF2BHW0245Y
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.064Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655251200053 maxt=1655272800000 ulid=01G5KGC7CMZBMZ3F9WB59XH5VX
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.064Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655272800065 maxt=1655294400000 ulid=01G5M4Z54HNS4WH3TEV58Y2DP0
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.065Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655294400027 maxt=1655316000000 ulid=01G5MSJ8XD0959V50JK46YPMEH
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.065Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655316000094 maxt=1655337600000 ulid=01G5NE5EX24V3EAPY8K397VS72
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.066Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655337600082 maxt=1655359200000 ulid=01G5P2RM6H86EG3S8WB62EARG6
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.066Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655359200029 maxt=1655380800000 ulid=01G5PQBVAD8PFDG3C63AG4ZVAK
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.067Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655380800018 maxt=1655402400000 ulid=01G5QBYWZ7DSMAJ33Z59S15PAY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.067Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655402400075 maxt=1655424000000 ulid=01G5R0J3EECE6EXART79PHG65S
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.067Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655424000005 maxt=1655445600000 ulid=01G5RN5ERF86JA4JS0VGR40T0R
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.068Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655445600001 maxt=1655467200000 ulid=01G5S9RDMFY0H57AEHV4FCE3NY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.069Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655467200037 maxt=1655488800000 ulid=01G5SYBMNJGX7RMJKX599SDBA4
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.069Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655488800046 maxt=1655510400000 ulid=01G5TJYXKPTKM9HDQNCR5E0N5S
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.070Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655510400048 maxt=1655532000000 ulid=01G5V7HYTCK7YHRRN0YY3P6NRZ
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.071Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655532000024 maxt=1655553600000 ulid=01G5VW5BW2FKAAKMV6S4E25TGA
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.071Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655553600050 maxt=1655575200000 ulid=01G5WGRH7TKASRJ4KC5BFFP6CW
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.072Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655575200011 maxt=1655596800000 ulid=01G5X5BMH51P8K5GPNN274FKKY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.073Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655596800053 maxt=1655618400000 ulid=01G5XSYTAS9DHZ8Z9K717BMGD7
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.074Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655618400034 maxt=1655640000000 ulid=01G5YEJ30TV86JSATFG3DPZGNY
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.075Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655640000011 maxt=1655661600000 ulid=01G5Z359YMR66GZHYEZWRGV553
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.075Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655661600041 maxt=1655683200000 ulid=01G5ZQRAXJJ7Y4139FC47C6R9Q
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.076Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655704800001 maxt=1655712000000 ulid=01G605E5Q66YYVD4EFBDJ0J1Q3
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.076Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655712000000 maxt=1655719200000 ulid=01G60C9WZCQTPRM2Z67ZKBCPFS
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.077Z caller=repair.go:57 component=tsdb msg="Found healthy block" mint=1655683200001 maxt=1655704800000 ulid=01G60CBFND7HYV9BWCBZMFJZ4K
Mon, 20 Jun 2022 13:38:33 | level=info ts=2022-06-20T11:38:33.877Z caller=head.go:780 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
Mon, 20 Jun 2022 13:38:36 | level=info ts=2022-06-20T11:38:36.079Z caller=head.go:794 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.20253601s
Mon, 20 Jun 2022 13:38:36 | level=info ts=2022-06-20T11:38:36.079Z caller=head.go:800 component=tsdb msg="Replaying WAL, this may take a while"
Mon, 20 Jun 2022 13:38:43 | level=info ts=2022-06-20T11:38:43.443Z caller=head.go:826 component=tsdb msg="WAL checkpoint loaded"
Mon, 20 Jun 2022 13:38:46 | level=info ts=2022-06-20T11:38:46.451Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16619 maxSegment=16630
Mon, 20 Jun 2022 13:38:49 | level=info ts=2022-06-20T11:38:49.473Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16620 maxSegment=16630
Mon, 20 Jun 2022 13:38:54 | level=info ts=2022-06-20T11:38:54.408Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16621 maxSegment=16630
Mon, 20 Jun 2022 13:38:55 | level=info ts=2022-06-20T11:38:55.863Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16622 maxSegment=16630
Mon, 20 Jun 2022 13:38:58 | level=info ts=2022-06-20T11:38:58.556Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16623 maxSegment=16630
Mon, 20 Jun 2022 13:39:01 | level=info ts=2022-06-20T11:39:01.472Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16624 maxSegment=16630
Mon, 20 Jun 2022 13:39:03 | level=info ts=2022-06-20T11:39:03.790Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16625 maxSegment=16630
Mon, 20 Jun 2022 13:39:06 | level=info ts=2022-06-20T11:39:06.553Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16626 maxSegment=16630
Mon, 20 Jun 2022 13:39:11 | level=info ts=2022-06-20T11:39:11.685Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16627 maxSegment=16630
Mon, 20 Jun 2022 13:39:14 | level=info ts=2022-06-20T11:39:14.656Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16628 maxSegment=16630
Mon, 20 Jun 2022 13:39:16 | level=info ts=2022-06-20T11:39:16.662Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16629 maxSegment=16630
Mon, 20 Jun 2022 13:39:16 | level=info ts=2022-06-20T11:39:16.662Z caller=head.go:854 component=tsdb msg="WAL segment loaded" segment=16630 maxSegment=16630
Mon, 20 Jun 2022 13:39:16 | level=info ts=2022-06-20T11:39:16.662Z caller=head.go:860 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=7.364090293s wal_replay_duration=33.219008395s total_replay_duration=42.7856811s
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.613Z caller=main.go:851 fs_type=EXT4_SUPER_MAGIC
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.613Z caller=main.go:854 msg="TSDB started"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.613Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Starting WAL watcher" queue=742148
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Starting scraped metadata watcher"
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Replaying WAL" queue=742148
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.620Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.621Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.621Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.622Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.622Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.623Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.623Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.624Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.625Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.625Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.626Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.626Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.627Z caller=kubernetes.go:282 component="discovery manager notify" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.710Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=96.37768ms remote_storage=340.721µs web_handler=434ns query_engine=540ns scrape=209.884µs scrape_sd=7.039296ms notify=20.722µs notify_sd=22.71941ms rules=60.15021ms
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.710Z caller=main.go:796 msg="Server is ready to receive web requests."
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.710Z caller=main.go:981 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.716Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | ts=2022-06-20T11:39:18.716Z caller=level.go:63 component=k8s_client_runtime level=error func=ErrorDepth msg="pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: Get \"https://10.43.0.1:443/api/v1/namespaces/sealed-services/pods?allowWatchBookmarks=true&resourceVersion=495641055&timeout=6m49s&timeoutSeconds=409&watch=true\": context canceled"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.717Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.718Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.718Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.719Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.719Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.720Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.720Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.721Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.721Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.722Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.722Z caller=kubernetes.go:282 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.723Z caller=kubernetes.go:282 component="discovery manager notify" discovery=kubernetes msg="Using pod service account via in-cluster config"
Mon, 20 Jun 2022 13:39:18 | level=info ts=2022-06-20T11:39:18.835Z caller=main.go:1012 msg="Completed loading of configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml totalDuration=125.168082ms remote_storage=73.181µs web_handler=443ns query_engine=589ns scrape=39.52µs scrape_sd=6.998634ms notify=14.04µs notify_sd=777.09µs rules=111.319021ms
Mon, 20 Jun 2022 13:39:32 | ts=2022-06-20T11:39:32.987Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Done replaying WAL" duration=14.367819655s
Mon, 20 Jun 2022 13:39:40 | ts=2022-06-20T11:39:40.136Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=1 to=6
Mon, 20 Jun 2022 13:39:58 | ts=2022-06-20T11:39:58.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=6 to=10
Mon, 20 Jun 2022 13:40:08 | ts=2022-06-20T11:40:08.622Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=10 to=14
Mon, 20 Jun 2022 13:40:18 | ts=2022-06-20T11:40:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=14 to=19
Mon, 20 Jun 2022 13:40:28 | ts=2022-06-20T11:40:28.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=19 to=26
Mon, 20 Jun 2022 13:40:38 | ts=2022-06-20T11:40:38.621Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:40:48 | ts=2022-06-20T11:40:48.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=26 to=43
Mon, 20 Jun 2022 13:41:08 | ts=2022-06-20T11:41:08.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=43 to=66
Mon, 20 Jun 2022 13:41:28 | ts=2022-06-20T11:41:28.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=66 to=97
Mon, 20 Jun 2022 13:41:48 | ts=2022-06-20T11:41:48.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:41:58 | ts=2022-06-20T11:41:58.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=97 to=172
Mon, 20 Jun 2022 13:42:48 | ts=2022-06-20T11:42:48.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=172 to=348
Mon, 20 Jun 2022 13:43:18 | ts=2022-06-20T11:43:18.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:43:48 | ts=2022-06-20T11:43:48.620Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:43:48 | ts=2022-06-20T11:43:48.620Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="non-recoverable error" count=100 exemplarCount=0 err="context canceled"
Mon, 20 Jun 2022 13:43:48 | ts=2022-06-20T11:43:48.621Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to flush all samples on shutdown" count=1003390020
Mon, 20 Jun 2022 13:43:58 | ts=2022-06-20T11:43:58.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=348 to=802
Mon, 20 Jun 2022 13:44:28 | ts=2022-06-20T11:44:28.620Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Currently resharding, skipping."
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.620Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.620Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="non-recoverable error" count=100 exemplarCount=0 err="context canceled"
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.622Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:44:58 | ts=2022-06-20T11:44:58.622Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to flush all samples on shutdown" count=3274907571
Mon, 20 Jun 2022 13:45:08 | ts=2022-06-20T11:45:08.632Z caller=dedupe.go:112 component=remote level=info remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Remote storage resharding" from=802 to=1000
Mon, 20 Jun 2022 13:46:09 | ts=2022-06-20T11:46:09.029Z caller=dedupe.go:112 component=remote level=warn remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to send batch, retrying" err="Post \"http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push\": context canceled"
Mon, 20 Jun 2022 13:46:09 | ts=2022-06-20T11:46:09.029Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="non-recoverable error" count=100 exemplarCount=0 err="context canceled"
Mon, 20 Jun 2022 13:46:09 | ts=2022-06-20T11:46:09.316Z caller=dedupe.go:112 component=remote level=error remote_name=742148 url=http://mimir-distributed-nginx.cattle-monitoring-system:80/api/v1/push msg="Failed to flush all samples on shutdown" count=2876743850

@ghost
Copy link
Author

ghost commented Jun 20, 2022

Config:

admin_api:
  affinity: {}
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
alertmanager:
  affinity: {}
  annotations: {}
  enabled: true
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 10Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  statefulSet:
    enabled: true
  statefulStrategy:
    type: RollingUpdate
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
compactor:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - compactor
            topologyKey: kubernetes.io/hostname
          weight: 100
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 90Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    type: RollingUpdate
  terminationGracePeriodSeconds: 240
  tolerations: []
distributor:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - distributor
          topologyKey: kubernetes.io/hostname
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
enterprise:
  enabled: false
  image:
    repository: grafana/enterprise-metrics
    tag: v2.1.0
  legacyLabels: false
externalConfigSecretName: '{{ include "mimir.resourceName" (dict "ctx" . "component" "config") }}'
externalConfigVersion: '0'
fullnameOverride: null
gateway:
  affinity: {}
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  ingress:
    annotations: {}
    enabled: false
    hosts:
      - host: gateway.gem.example.com
        paths:
          - path: /
    tls:
      - hosts:
          - gateway.gem.example.com
        secretName: gem-gateway-tls
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
    port: null
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
  useDefaultProxyURLs: true
global:
  clusterDomain: cluster.local
  dnsNamespace: kube-system
  dnsService: kube-dns
image:
  pullPolicy: IfNotPresent
  repository: grafana/mimir
  tag: 2.1.0
ingester:
  affinity: {}
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 90Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget:
    maxUnavailable: 1
  podLabels: {}
  podManagementPolicy: Parallel
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60
  replicas: 3
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  statefulSet:
    enabled: true
  statefulStrategy:
    type: RollingUpdate
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 240
  tolerations: []
license:
  contents: NOTAVALIDLICENSE
  external: false
  secretName: '{{ include "mimir.resourceName" (dict "ctx" . "component" "license") }}'
memcached:
  architecture: high-availability
  arguments:
    - '-m 8192'
    - '-o'
    - modern
    - '-v'
    - '-I 1m'
    - '-c 4096'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '1048576'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 9830Mi
    requests:
      cpu: 500m
      memory: 9830Mi
memcached-metadata:
  architecture: high-availability
  arguments:
    - '-m 512'
    - '-o'
    - modern
    - '-v'
    - '-I 1m'
    - '-c 1024'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '1048576'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 614Mi
    requests:
      cpu: 500m
      memory: 614Mi
memcached-queries:
  architecture: high-availability
  arguments:
    - '-m 2048'
    - '-o'
    - modern
    - '-v'
    - '-I 15m'
    - '-c 1024'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '15728640'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 2457Mi
    requests:
      cpu: 500m
      memory: 2457Mi
memcached-results:
  architecture: high-availability
  arguments:
    - '-m 512'
    - '-o'
    - modern
    - '-v'
    - '-I 1m'
    - '-c 1024'
  enabled: false
  image:
    repository: memcached
    tag: 1.6.9
  maxItemMemory: '1048576'
  metrics:
    enabled: true
    image:
      registry: quay.io
      repository: prometheus/memcached-exporter
      tag: v0.9.0
  replicaCount: 1
  resources:
    limits:
      memory: 614Mi
    requests:
      cpu: 500m
      memory: 614Mi
mimir:
  config: >
    {{- if not .Values.enterprise.enabled -}}

    multitenancy_enabled: false

    {{- end }}


    limits:
      compactor_blocks_retention_period: 604800s
      ingestion_rate: 60000
      max_global_series_per_metric: 2000000
      max_global_series_per_user: 2000000
      max_label_names_per_series: 60

    distributor:
      instance_limits:
        max_ingestion_rate: 0
        max_inflight_push_requests: 0
      remote_timeout: 30s


    activity_tracker:
      filepath: /data/metrics-activity.log

    alertmanager:
      data_dir: '/data'
      enable_api: true
      external_url: '/alertmanager'

    {{- if .Values.minio.enabled }}

    alertmanager_storage:
      backend: s3
      s3:
        endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
        bucket_name: {{ include "mimir.minioBucketPrefix" . }}-ruler
        access_key_id: {{ .Values.minio.accessKey }}
        secret_access_key: {{ .Values.minio.secretKey }}
        insecure: true
    {{- end }}


    frontend_worker:
      frontend_address: {{ template "mimir.fullname" . }}-query-frontend-headless.{{ .Release.Namespace }}.svc:{{ include "mimir.serverGrpcListenPort" . }}

    ruler:
      enable_api: true
      rule_path: '/data'
      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.{{ template "mimir.fullname" . }}-alertmanager-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}/alertmanager

    server:
      grpc_server_max_recv_msg_size: 104857600
      grpc_server_max_send_msg_size: 104857600
      grpc_server_max_concurrent_streams: 1000

    frontend:
      log_queries_longer_than: 10s
      align_queries_with_step: true
      {{- if index .Values "memcached-results" "enabled" }}
      results_cache:
        backend: memcached
        memcached:
          addresses: dns+{{ .Release.Name }}-memcached-results.{{ .Release.Namespace }}.svc:11211
          max_item_size: {{ (index .Values "memcached-results").maxItemMemory }}
      cache_results: true
      {{- end }}

    compactor:
      data_dir: "/data"

    ingester:
      ring:
        final_sleep: 0s
        num_tokens: 512
        unregister_on_shutdown: false

    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600

    runtime_config:
      file: /var/{{ include "mimir.name" . }}/runtime.yaml

    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      join_members:
      - {{ include "mimir.fullname" . }}-gossip-ring

    # This configures how the store-gateway synchronizes blocks stored in the
    bucket. It uses Minio by default for getting started (configured via flags)
    but this should be changed for production deployments.

    blocks_storage:
      backend: s3
      tsdb:
        dir: /data/tsdb
      bucket_store:
        sync_dir: /data/tsdb-sync
        {{- if .Values.memcached.enabled }}
        chunks_cache:
          backend: memcached
          memcached:
            addresses: dns+{{ .Release.Name }}-memcached.{{ .Release.Namespace }}.svc:11211
            max_item_size: {{ .Values.memcached.maxItemMemory }}
        {{- end }}
        {{- if index .Values "memcached-metadata" "enabled" }}
        metadata_cache:
          backend: memcached
          memcached:
            addresses: dns+{{ .Release.Name }}-memcached-metadata.{{ .Release.Namespace }}.svc:11211
            max_item_size: {{ (index .Values "memcached-metadata").maxItemMemory }}
        {{- end }}
        {{- if index .Values "memcached-queries" "enabled" }}
        index_cache:
          backend: memcached
          memcached:
            addresses: dns+{{ .Release.Name }}-memcached-queries.{{ .Release.Namespace }}.svc:11211
            max_item_size: {{ (index .Values "memcached-queries").maxItemMemory }}
        {{- end }}
      {{- if .Values.minio.enabled }}
      s3:
        endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
        bucket_name: {{ include "mimir.minioBucketPrefix" . }}-tsdb
        access_key_id: {{ .Values.minio.accessKey }}
        secret_access_key: {{ .Values.minio.secretKey }}
        insecure: true
      {{- end }}

    {{- if .Values.minio.enabled }}

    ruler_storage:
      backend: s3
      s3:
        endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
        bucket_name: {{ include "mimir.minioBucketPrefix" . }}-ruler
        access_key_id: {{ .Values.minio.accessKey }}
        secret_access_key: {{ .Values.minio.secretKey }}
        insecure: true
    {{- end }}


    {{- if .Values.enterprise.enabled }}

    multitenancy_enabled: true


    admin_api:
      leader_election:
        enabled: true
        ring:
          kvstore:
            store: "memberlist"

    {{- if .Values.minio.enabled }}

    admin_client:
      storage:
        type: s3
        s3:
          endpoint: {{ .Release.Name }}-minio.{{ .Release.Namespace }}.svc:9000
          bucket_name: enterprise-metrics-admin
          access_key_id: {{ .Values.minio.accessKey }}
          secret_access_key: {{ .Values.minio.secretKey }}
          insecure: true
    {{- end }}


    auth:
      type: enterprise

    cluster_name: "{{ .Release.Name }}"


    license:
      path: "/license/license.jwt"

    {{- if .Values.gateway.useDefaultProxyURLs }}

    gateway:
      proxy:
        default:
          url: http://{{ template "mimir.fullname" . }}-admin-api.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        admin_api:
          url: http://{{ template "mimir.fullname" . }}-admin-api.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        alertmanager:
          url: http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        compactor:
          url: http://{{ template "mimir.fullname" . }}-compactor.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        distributor:
          url: dns:///{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverGrpcListenPort" . }}
        ingester:
          url: http://{{ template "mimir.fullname" . }}-ingester.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        query_frontend:
          url: http://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        ruler:
          url: http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
        store_gateway:
          url: http://{{ template "mimir.fullname" . }}-store-gateway.{{ .Release.Namespace }}.svc:{{ include "mimir.serverHttpListenPort" . }}
    {{- end }}


    instrumentation:
      enabled: true
      distributor_client:
        address: 'dns:///{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverGrpcListenPort" . }}'

    {{- end }}
minio:
  accessKey: grafana-mimir
  buckets:
    - name: mimir-tsdb
      policy: none
      purge: false
    - name: mimir-ruler
      policy: none
      purge: false
    - name: enterprise-metrics-tsdb
      policy: none
      purge: false
    - name: enterprise-metrics-admin
      policy: none
      purge: false
    - name: enterprise-metrics-ruler
      policy: none
      purge: false
  enabled: true
  persistence:
    size: 270Gi
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  secretKey: supersecret
nameOverride: null
nginx:
  affinity: |
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: component
                  operator: In
                  values:
                    - nginx
            topologyKey: failure-domain.beta.kubernetes.io/zone
  annotations: {}
  autoscaling:
    enabled: false
    maxReplicas: 3
    minReplicas: 1
    targetCPUUtilizationPercentage: 60
    targetMemoryUtilizationPercentage: null
  basicAuth:
    enabled: false
    existingSecret: null
    htpasswd: >-
      {{ htpasswd (required "'nginx.basicAuth.username' is required"
      .Values.nginx.basicAuth.username) (required "'nginx.basicAuth.password' is
      required" .Values.nginx.basicAuth.password) }}
    password: null
    username: null
  containerSecurityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    readOnlyRootFilesystem: true
  deploymentStrategy:
    type: RollingUpdate
  enabled: true
  extraArgs: []
  extraEnv: []
  extraEnvFrom: []
  extraVolumeMounts: []
  extraVolumes: []
  image:
    pullPolicy: IfNotPresent
    registry: docker.io
    repository: nginxinc/nginx-unprivileged
    tag: 1.19-alpine
  ingress:
    annotations: {}
    enabled: false
    hosts:
      - host: nginx.loki.example.com
        paths:
          - path: /
    tls:
      - hosts:
          - nginx.loki.example.com
        secretName: loki-nginx-tls
  nginxConfig:
    file: |
      worker_processes  5;  ## Default: 1
      error_log  /dev/stderr;
      pid        /tmp/nginx.pid;
      worker_rlimit_nofile 8192;

      events {
        worker_connections  4096;  ## Default: 1024
      }

      http {
        client_body_temp_path /tmp/client_temp;
        proxy_temp_path       /tmp/proxy_temp_path;
        fastcgi_temp_path     /tmp/fastcgi_temp;
        uwsgi_temp_path       /tmp/uwsgi_temp;
        scgi_temp_path        /tmp/scgi_temp;

        default_type application/octet-stream;
        log_format   {{ .Values.nginx.nginxConfig.logFormat }}

        {{- if .Values.nginx.verboseLogging }}
        access_log   /dev/stderr  main;
        {{- else }}

        map $status $loggable {
          ~^[23]  0;
          default 1;
        }
        access_log   /dev/stderr  main  if=$loggable;
        {{- end }}

        sendfile     on;
        tcp_nopush   on;
        resolver {{ .Values.global.dnsService }}.{{ .Values.global.dnsNamespace }}.svc.{{ .Values.global.clusterDomain }};

        {{- with .Values.nginx.nginxConfig.httpSnippet }}
        {{ . | nindent 2 }}
        {{- end }}

        server {
          listen 8080;

          {{- if .Values.nginx.basicAuth.enabled }}
          auth_basic           "Mimir";
          auth_basic_user_file /etc/nginx/secrets/.htpasswd;
          {{- end }}

          location = / {
            return 200 'OK';
            auth_basic off;
          }

          {{- if not (include "mimir.calculatedConfig" . | fromYaml).multitenancy_enabled }}
          proxy_set_header X-Scope-OrgID 0;
          {{- end }}

          # Distributor endpoints
          location /distributor {
            proxy_pass      http://{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /api/v1/push {
            proxy_pass      http://{{ template "mimir.fullname" . }}-distributor-headless.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Alertmanager endpoints
          location {{ template "mimir.alertmanagerHttpPrefix" . }} {
            proxy_pass      http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /multitenant_alertmanager/status {
            proxy_pass      http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /api/v1/alerts {
            proxy_pass      http://{{ template "mimir.fullname" . }}-alertmanager.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Ruler endpoints
          location {{ template "mimir.prometheusHttpPrefix" . }}/config/v1/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location {{ template "mimir.prometheusHttpPrefix" . }}/api/v1/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          location /api/v1/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location {{ template "mimir.prometheusHttpPrefix" . }}/api/v1/alerts {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location {{ template "mimir.prometheusHttpPrefix" . }}/rules {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }
          location = /ruler/ring {
            proxy_pass      http://{{ template "mimir.fullname" . }}-ruler.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Rest of {{ template "mimir.prometheusHttpPrefix" . }} goes to the query frontend
          location {{ template "mimir.prometheusHttpPrefix" . }} {
            proxy_pass      http://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          # Buildinfo endpoint can go to any component
          location = /api/v1/status/buildinfo {
            proxy_pass      http://{{ template "mimir.fullname" . }}-query-frontend.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.serverHttpListenPort" . }}$request_uri;
          }

          {{- with .Values.nginx.nginxConfig.serverSnippet }}
          {{ . | nindent 4 }}
          {{- end }}
        }
      }
    httpSnippet: ''
    logFormat: |-
      main '$remote_addr - $remote_user [$time_local]  $status '
              '"$request" $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for"';
    serverSnippet: ''
  nodeSelector: {}
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  podSecurityContext:
    fsGroup: 101
    runAsGroup: 101
    runAsNonRoot: true
    runAsUser: 101
  priorityClassName: null
  readinessProbe:
    httpGet:
      path: /
      port: http-metric
    initialDelaySeconds: 15
    timeoutSeconds: 1
  replicas: 1
  resources: {}
  service:
    annotations: {}
    clusterIP: null
    labels: {}
    loadBalancerIP: null
    nodePort: null
    port: 80
    type: ClusterIP
  terminationGracePeriodSeconds: 30
  tolerations: []
  verboseLogging: true
overrides_exporter:
  affinity: {}
  annotations: {}
  enabled: true
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  livenessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 60
  tolerations: []
querier:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - querier
            topologyKey: kubernetes.io/hostname
          weight: 100
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 2
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 180
  tolerations: []
query_frontend:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: target
                  operator: In
                  values:
                    - query-frontend
            topologyKey: kubernetes.io/hostname
          weight: 100
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 180
  tolerations: []
rbac:
  pspEnabled: true
ruler:
  affinity: {}
  annotations: {}
  enabled: true
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistence:
    subPath: null
  podAnnotations: {}
  podDisruptionBudget: {}
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 45
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  terminationGracePeriodSeconds: 180
  tolerations: []
runtimeConfig: {}
serviceAccount:
  annotations: {}
  create: true
  name: null
serviceMonitor:
  annotations: {}
  enabled: false
  interval: null
  labels: {}
  namespace: null
  namespaceSelector: {}
  relabelings: []
  scheme: http
  scrapeTimeout: null
  tlsConfig: null
store_gateway:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - store-gateway
          topologyKey: kubernetes.io/hostname
  annotations: {}
  env: []
  extraArgs: {}
  extraContainers: []
  extraVolumeMounts: []
  extraVolumes: []
  initContainers: []
  nodeSelector: {}
  persistentVolume:
    accessModes:
      - ReadWriteOnce
    annotations: {}
    enabled: true
    size: 90Gi
    subPath: ''
  podAnnotations: {}
  podDisruptionBudget:
    maxUnavailable: 1
  podLabels: {}
  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 60
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
  securityContext: {}
  service:
    annotations: {}
    labels: {}
  strategy:
    type: RollingUpdate
  terminationGracePeriodSeconds: 240
  tolerations: []
tokengenJob:
  annotations: {}
  enable: true
  env: []
  extraArgs: {}
  initContainers: []
useExternalConfig: false

@krajorama
Copy link
Contributor

Hi, out-of-order sample support is an upcoming feature in Mimir (grafana/mimir#2187). Possibly @codesome can comment on this issue?

@codesome
Copy link
Member

Mimir 2.2 will have experimental support for accepting out of order samples. The release will come with docs on how to enable this feature and use this. Stay tuned!

@ghost
Copy link
Author

ghost commented Jul 2, 2022

Very nice. But what about a alternative solution until then? How can i tell prometheus not to send old samples? How can i disable this behaviour? I know this is a question for another issue in another place but i am curious. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@krajorama @codesome