compactor: Thanos Compactor compaction level number being set to 93+ for 1 half of HA Prometheus (S3 Storage)

[jbarber-rbbn]

Ping @bwplotka

Thanos, Prometheus and Golang version used:
Thanos: 0.31.0
Prometheus: 2.41.0
Thanos-Sidecar: 0.30.1
Golang: go1.19.4

Image Tags Used:
Prometheus: bitnami/prometheus-operator:0.62.0-debian-11-r0
Thanos-Sidecar: bitnami/thanos:0.30.1-scratch-r0
Thanos: bitnami/thanos:0.31.0-scratch-r5

Object Storage Provider:
S3 on Ceph

What happened:
Running thanos tools bucket inspect '--selector=prometheus_replica="prometheus-prometheus-ha-kube-prometh-prometheus-1"' we see COMP-LEVEL set to numbers greater than expected values. (1,2,3, and 4 are expected)
We see 93, 129 - 153, and 168.

This occurs only on 1 half of the HA pair of prometheus that we run.
The other half of the HA pair does not have this problem.

What you expected to happen:
Compaction level does not exceed 4.

How to reproduce it (as minimally and precisely as possible):
Not sure.

Full logs to relevant components:

Thanos Bucket Inspect Logs

level=info ts=2024-06-04T20:32:02.405143949Z caller=factory.go:52 msg="loading bucket configuration"
level=info ts=2024-06-04T20:32:02.511419755Z caller=fetcher.go:478 component=block.BaseFetcher msg="successfully synchronized block metadata" duration=105.954942ms duration_ms=105 cached=271 returned=271 partial=0
|            ULID            |         FROM         |        UNTIL         |     RANGE      |   UNTIL-DOWN    |   #SERIES   |    #SAMPLES    |   #CHUNKS   | COMP-LEVEL | COMP-FAILED |
                                              LABELS                                                                         | RESOLUTION |  SOURCE   |
|----------------------------|----------------------|----------------------|----------------|-----------------|-------------|----------------|-------------|------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|------------|-----------|
| 01GSQ3G5MMGD2WWC7C4GHQN732 | 2023-02-02T00:00:01Z | 2023-02-16T00:00:00Z | 335h59m58.333s | -               | 104,671,400 | 185,020,432    | 105,130,624 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01GTJ7XN17NFF81DPXEVM7F47V | 2023-02-16T00:00:00Z | 2023-03-02T00:00:00Z | 335h59m59.698s | -               | 200,607,563 | 309,929,191    | 201,057,395 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01GVN9B9CM2BZJ7WBRMBF634SV | 2023-03-02T00:00:00Z | 2023-03-16T00:00:00Z | 335h59m59.643s | -               | 161,651,027 | 260,699,132    | 162,018,035 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01GWYCACA1VQEWA41EW4WC2HS3 | 2023-03-16T00:00:00Z | 2023-03-30T00:00:00Z | 336h0m0s       | -               | 231,561,083 | 363,392,700    | 231,833,539 | 5          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01GY1Y4R3E859NXGW0J03PDHQ5 | 2023-03-30T00:00:00Z | 2023-04-09T00:00:00Z | 240h0m0s       | -               | 204,211,982 | 311,053,436    | 204,736,552 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01GZ1Z3B3JCYJQH4BR979Q6WCE | 2023-04-13T00:00:00Z | 2023-04-27T00:00:00Z | 336h0m0s       | -               | 202,253,150 | 305,874,880    | 202,837,493 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H063F4H6W63YAF5VSQC7VC9A | 2023-04-27T00:00:00Z | 2023-05-11T00:00:00Z | 336h0m0s       | -               | 197,188,234 | 308,036,081    | 197,652,819 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H1ACAS5JPWM65CSY0QN8540H | 2023-05-11T00:00:00Z | 2023-05-25T00:00:00Z | 336h0m0s       | -               | 114,533,593 | 239,441,617    | 115,026,485 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H2GGVNH6HSD3BHMPNE4SN1PV | 2023-05-25T00:00:00Z | 2023-06-08T00:00:00Z | 336h0m0s       | -96h0m0s        | 38,339,699  | 959,915,260    | 45,678,617  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H2GNRM8PEW9XAD6R7W47275K | 2023-05-25T00:00:00Z | 2023-06-08T00:00:00Z | 336h0m0s       | -               | 38,339,699  | 115,087,972    | 38,590,833  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H3H2DB6J7AADFJ32QYSXS0VJ | 2023-06-08T00:00:00Z | 2023-06-22T00:00:00Z | 335h59m59.899s | -95h59m59.899s  | 87,432,858  | 1,389,519,080  | 96,230,533  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H3H6V6Z133KPJSYDCZ6G24VW | 2023-06-08T00:00:00Z | 2023-06-22T00:00:00Z | 335h59m59.899s | -               | 87,432,858  | 200,963,235    | 87,937,144  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H4MTA190K0CJ6X9H6EFPSGMB | 2023-06-22T00:00:00Z | 2023-07-06T00:00:00Z | 336h0m0s       | -96h0m0s        | 60,395,409  | 1,384,820,966  | 69,153,222  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H4N0XGRT658F3MVSJ269N2TK | 2023-06-22T00:00:00Z | 2023-07-06T00:00:00Z | 336h0m0s       | -               | 60,395,409  | 171,273,927    | 60,896,048  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H5S4VVB8406T7QHRZYBYGX3Z | 2023-07-06T00:00:00Z | 2023-07-20T00:00:00Z | 335h59m59.989s | -95h59m59.989s  | 60,560,242  | 1,231,478,732  | 69,170,547  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H5S8GPTCHT749WW9ZVATJQD9 | 2023-07-06T00:00:00Z | 2023-07-20T00:00:00Z | 335h59m59.989s | -               | 60,560,242  | 158,776,940    | 61,053,951  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H779VP6H96S7H10E1MMVRD7Y | 2023-07-20T00:00:00Z | 2023-08-03T00:00:00Z | 335h59m59.981s | -95h59m59.981s  | 88,648,352  | 1,638,690,648  | 101,527,766 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H77DYVM0JJDRVHP963QG78T6 | 2023-07-20T00:00:00Z | 2023-08-03T00:00:00Z | 335h59m59.981s | -               | 88,648,352  | 218,553,600    | 89,475,269  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H81BGPBBAA1GAXFQ55GDTSYB | 2023-08-03T00:00:00Z | 2023-08-17T00:00:00Z | 336h0m0s       | -96h0m0s        | 101,127,790 | 1,814,615,808  | 112,830,964 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H81PPD8SW48REZZ2CACTKTQ5 | 2023-08-03T00:00:00Z | 2023-08-17T00:00:00Z | 336h0m0s       | -               | 101,127,790 | 244,745,870    | 102,027,825 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01H959GZZWQFCRRWKEX2QNAHXJ | 2023-08-17T00:00:00Z | 2023-08-31T00:00:00Z | 336h0m0s       | -96h0m0s        | 132,184,971 | 1,762,011,911  | 143,195,125 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01H95R9G1RZ8WTNY36ETK1MZYJ | 2023-08-17T00:00:00Z | 2023-08-31T00:00:00Z | 336h0m0s       | -               | 132,184,971 | 268,882,018    | 132,756,367 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HA9KF7SR12ZHWW0XA0KFYA2R | 2023-08-31T00:00:00Z | 2023-09-14T00:00:00Z | 336h0m0s       | -96h0m0s        | 177,854,728 | 1,912,228,443  | 190,179,077 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HAA72XNM09YN2ZE395YB9XGQ | 2023-08-31T00:00:00Z | 2023-09-14T00:00:00Z | 336h0m0s       | -               | 177,854,728 | 323,969,304    | 178,632,645 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HBDJFBHXE3RJNEFXM4XP7Z9Q | 2023-09-14T00:00:00Z | 2023-09-28T00:00:00Z | 336h0m0s       | -96h0m0s        | 121,456,019 | 1,477,769,927  | 130,849,889 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HBDZE6P8W5PDJXK23FT4KR8D | 2023-09-14T00:00:00Z | 2023-09-28T00:00:00Z | 336h0m0s       | -               | 121,456,019 | 235,267,883    | 122,188,508 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HCHBBXA50STS4446C861T2PD | 2023-09-28T00:00:00Z | 2023-10-12T00:00:00Z | 336h0m0s       | -96h0m0s        | 94,589,375  | 1,464,198,010  | 103,959,433 | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HCHNZQS99ZBAG1T71TSGK3RN | 2023-09-28T00:00:00Z | 2023-10-12T00:00:00Z | 336h0m0s       | -               | 94,589,375  | 210,295,277    | 95,461,737  | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HDN7V01GE177RBY89JK0FQ57 | 2023-10-12T00:00:01Z | 2023-10-26T00:00:00Z | 335h59m58.333s | -95h59m58.333s  | 245,323     | 687,627,830    | 5,441,653   | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HDN8JGD9BN1H7WRXJJFRPFJA | 2023-10-12T00:00:01Z | 2023-10-26T00:00:00Z | 335h59m58.333s | -               | 245,323     | 57,778,384     | 737,066     | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HES9B8VHBY5BJ8CWM65RC2G6 | 2023-10-26T00:00:01Z | 2023-11-09T00:00:00Z | 335h59m58.565s | -95h59m58.565s  | 296,237     | 678,872,177    | 5,561,016   | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HESA20EEGKTENMJBAQ3JA4DG | 2023-10-26T00:00:01Z | 2023-11-09T00:00:00Z | 335h59m58.565s | -               | 296,237     | 57,364,701     | 770,145     | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HFXAWEYMWQWVJANCJNE8HEG1 | 2023-11-09T00:00:01Z | 2023-11-23T00:00:00Z | 335h59m58.564s | -95h59m58.564s  | 245,019     | 654,462,779    | 5,022,391   | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HFXBJ4P06GACWDY18K96XQ0Q | 2023-11-09T00:00:01Z | 2023-11-23T00:00:00Z | 335h59m58.564s | -               | 245,019     | 55,003,592     | 745,392     | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HH1CB9HATP5MTK1ESZS82QKY | 2023-11-23T00:00:01Z | 2023-12-07T00:00:00Z | 335h59m58.564s | -95h59m58.564s  | 261,635     | 617,401,022    | 4,863,345   | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HH1CNPPF4WH6HZ98TXNZV49X | 2023-11-23T00:00:01Z | 2023-12-07T00:00:00Z | 335h59m58.564s | -               | 261,635     | 51,963,234     | 683,662     | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HJ5DTCGFHAS8J092WVCTHWR2 | 2023-12-07T00:00:00Z | 2023-12-21T00:00:00Z | 335h59m59.412s | -95h59m59.412s  | 306,458     | 590,471,046    | 5,561,098   | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HJ5E45W1HD5V3MVYYYWXCAF0 | 2023-12-07T00:00:00Z | 2023-12-21T00:00:00Z | 335h59m59.412s | -               | 306,458     | 50,081,491     | 679,473     | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HK9FBXHNKEWPSARKX2W9EBAG | 2023-12-21T00:00:00Z | 2024-01-04T00:00:00Z | 335h59m59.214s | -95h59m59.214s  | 234,204     | 617,825,278    | 4,697,765   | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HK9FPN5HD88G85JWY609FQ7W | 2023-12-21T00:00:00Z | 2024-01-04T00:00:00Z | 335h59m59.214s | -               | 234,204     | 51,926,918     | 667,019     | 4          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HMD34B6EMSQHD79GKJGW3NE3 | 2024-01-04T00:00:00Z | 2024-01-18T00:00:00Z | 335h59m59.214s | -95h59m59.214s  | 260,639     | 585,794,074    | 4,926,093   | 93         | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HMD3DSB23408C84XMHB20QSS | 2024-01-04T00:00:00Z | 2024-01-18T00:00:00Z | 335h59m59.214s | -               | 260,639     | 49,350,020     | 675,704     | 93         | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HNH57R3MXB17Y2CG69V41ZEB | 2024-01-18T00:00:00Z | 2024-02-01T00:00:00Z | 335h59m59.214s | -95h59m59.214s  | 4,648,209   | 1,865,148,614  | 17,207,839  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HNH6DHEF1E6N80WTC0DJW8EE | 2024-01-18T00:00:00Z | 2024-02-01T00:00:00Z | 335h59m59.214s | -               | 4,648,209   | 160,751,821    | 6,068,460   | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HPN7BXNKA5ZEHV0XNRWJSE0D | 2024-02-01T00:00:00Z | 2024-02-15T00:00:00Z | 335h59m59.983s | -95h59m59.983s  | 7,685,718   | 2,795,338,465  | 27,718,861  | 166        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HPN95ZJBX6JEQ2GGHGWKC9T2 | 2024-02-01T00:00:00Z | 2024-02-15T00:00:00Z | 335h59m59.983s | -               | 7,685,718   | 241,854,941    | 9,445,819   | 166        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HQS908QF8HCCJEJAG40RBEWJ | 2024-02-15T00:00:00Z | 2024-02-29T00:00:00Z | 335h59m59.998s | -95h59m59.998s  | 8,876,800   | 2,933,372,409  | 29,003,630  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HQSAZF6XN72ATZD3NJMPNFWW | 2024-02-15T00:00:00Z | 2024-02-29T00:00:00Z | 335h59m59.998s | -               | 8,876,800   | 253,163,522    | 10,766,868  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HRXAP8R29C0K1SW3F9ZFNGYN | 2024-02-29T00:00:00Z | 2024-03-14T00:00:00Z | 336h0m0s       | -96h0m0s        | 11,772,664  | 3,080,639,014  | 32,560,550  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HRXCQ0Y4KP66JRTH2GTRTE4P | 2024-02-29T00:00:00Z | 2024-03-14T00:00:00Z | 336h0m0s       | -               | 11,772,664  | 268,441,923    | 13,688,047  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HT1C5NPV525Q9NNB5GZV7MWC | 2024-03-14T00:00:00Z | 2024-03-28T00:00:00Z | 335h59m59.998s | -95h59m59.998s  | 12,909,290  | 3,057,809,858  | 33,558,997  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HT1E485ZZ60EKX37ESXBN8RT | 2024-03-14T00:00:00Z | 2024-03-28T00:00:00Z | 335h59m59.998s | -               | 12,909,290  | 268,258,539    | 14,474,051  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HV5DKJKKM293YQGF737TXH9J | 2024-03-28T00:00:00Z | 2024-04-11T00:00:00Z | 335h59m59.992s | -95h59m59.992s  | 10,187,738  | 2,920,443,732  | 30,374,216  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HV5FE3JW3NAA1J45M5BE4918 | 2024-03-28T00:00:00Z | 2024-04-11T00:00:00Z | 335h59m59.992s | -               | 10,187,738  | 253,421,280    | 12,084,037  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HW9G307NK4P2NKVDB6BTA9QQ | 2024-04-11T00:00:00Z | 2024-04-25T00:00:00Z | 335h59m59.998s | -95h59m59.998s  | 8,893,993   | 2,946,417,645  | 29,188,475  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HW9KXBTZGFS958N3J18SWHX9 | 2024-04-11T00:00:00Z | 2024-04-25T00:00:00Z | 335h59m59.998s | -               | 8,893,993   | 254,635,218    | 10,888,684  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HXDFX4T0VRAFYT3YCPCDBK9E | 2024-04-25T00:00:00Z | 2024-05-09T00:00:00Z | 336h0m0s       | -296h0m0s       | 8,699,936   | 29,553,799,370 | 253,654,811 | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HXDGMTE2MXWYBMND5CD084CS | 2024-04-25T00:00:00Z | 2024-05-09T00:00:00Z | 336h0m0s       | -96h0m0s        | 8,699,923   | 2,965,585,693  | 29,197,974  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HXDJERADQ69PGS3FZF4VKV7W | 2024-04-25T00:00:00Z | 2024-05-09T00:00:00Z | 336h0m0s       | -               | 8,699,923   | 255,815,527    | 10,602,752  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HYHHG0BRGAMN1KQ8H68Q7H0M | 2024-05-09T00:00:00Z | 2024-05-23T00:00:00Z | 335h59m59.978s | -295h59m59.978s | 8,523,530   | 30,797,268,238 | 263,262,349 | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HYHKHAR02MV31PKGR72TMAVF | 2024-05-09T00:00:00Z | 2024-05-23T00:00:00Z | 335h59m59.978s | -95h59m59.978s  | 8,523,530   | 3,089,161,732  | 29,991,023  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HYHQPNH98BW2K8SDBF1EXBWB | 2024-05-09T00:00:00Z | 2024-05-23T00:00:00Z | 335h59m59.978s | -               | 8,523,530   | 266,143,453    | 10,548,377  | 168        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZD74RS7E2KBX11WXD8WA930 | 2024-05-23T00:00:00Z | 2024-06-02T18:00:00Z | 258h0m0s       | -218h0m0s       | 7,913,910   | 23,607,459,700 | 203,371,295 | 129        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZD7YGQZJ69305ZPCRPYFTTC | 2024-05-23T00:00:00Z | 2024-06-02T18:00:00Z | 258h0m0s       | -18h0m0s        | 7,913,896   | 2,368,929,737  | 23,751,608  | 129        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZD9G99G159S7G40P6V3NA0E | 2024-05-23T00:00:00Z | 2024-06-02T18:00:00Z | 258h0m0s       | -               | 7,913,896   | 205,066,619    | 8,712,697   | 129        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZDE02J8TD9C6SGKKEN00P4T | 2024-05-23T00:00:00Z | 2024-06-02T20:00:00Z | 260h0m0s       | -220h0m0s       | 7,954,892   | 23,787,527,919 | 204,913,267 | 130        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZDEQGPWQK1WNFZ4ZFST6N0R | 2024-05-23T00:00:00Z | 2024-06-02T20:00:00Z | 260h0m0s       | -20h0m0s        | 7,954,878   | 2,386,992,981  | 24,415,143  | 130        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZDG9N4W6G294XE0EF6DX6VE | 2024-05-23T00:00:00Z | 2024-06-02T20:00:00Z | 260h0m0s       | -               | 7,954,878   | 207,190,738    | 9,323,153   | 130        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZDMW4034WTWNEAQJJ7B31FF | 2024-05-23T00:00:00Z | 2024-06-02T22:00:00Z | 262h0m0s       | -222h0m0s       | 8,004,761   | 23,963,945,918 | 206,469,046 | 131        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZDNKTA64EDTZMFYK9MQN2BG | 2024-05-23T00:00:00Z | 2024-06-02T22:00:00Z | 262h0m0s       | -22h0m0s        | 8,004,747   | 2,404,809,987  | 24,598,099  | 131        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZDQ6HSZJBX026TYCM3RM03Z | 2024-05-23T00:00:00Z | 2024-06-02T22:00:00Z | 262h0m0s       | -               | 8,004,747   | 208,790,408    | 9,387,151   | 131        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZDVRPM168XZBRN1FA2B604R | 2024-05-23T00:00:00Z | 2024-06-03T00:00:00Z | 264h0m0s       | -224h0m0s       | 8,048,028   | 24,139,647,881 | 207,976,313 | 132        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZDWGH8CWP6YCYX5QKYHACTA | 2024-05-23T00:00:00Z | 2024-06-03T00:00:00Z | 264h0m0s       | -24h0m0s        | 8,048,014   | 2,422,429,168  | 24,670,345  | 132        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZDY2ZTFBT1XT7ZSZGW9AD92 | 2024-05-23T00:00:00Z | 2024-06-03T00:00:00Z | 264h0m0s       | -               | 8,048,014   | 209,867,277    | 9,433,366   | 132        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZE2MN2CFSVKE5MST93SD77T | 2024-05-23T00:00:00Z | 2024-06-03T02:00:00Z | 266h0m0s       | -226h0m0s       | 8,089,810   | 24,315,797,970 | 209,486,591 | 133        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZE3FF20TQ4N3N1VAF455QN8 | 2024-05-23T00:00:00Z | 2024-06-03T02:00:00Z | 266h0m0s       | -26h0m0s        | 8,089,796   | 2,440,098,767  | 24,726,357  | 133        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZE51SSZRVCPRB9AM2ECQNT3 | 2024-05-23T00:00:00Z | 2024-06-03T02:00:00Z | 266h0m0s       | -               | 8,089,796   | 211,745,575    | 9,483,291   | 133        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZE9GQP26C8C4WRQ4HKBHB22 | 2024-05-23T00:00:00Z | 2024-06-03T04:00:00Z | 268h0m0s       | -228h0m0s       | 8,150,455   | 24,491,731,593 | 211,016,503 | 134        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZEA8V2DWSBV5MMNSRE6DGCP | 2024-05-23T00:00:00Z | 2024-06-03T04:00:00Z | 268h0m0s       | -28h0m0s        | 8,150,441   | 2,457,758,004  | 24,818,572  | 134        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZEBW6XWP86V90449WF58JF6 | 2024-05-23T00:00:00Z | 2024-06-03T04:00:00Z | 268h0m0s       | -               | 8,150,441   | 213,364,873    | 9,549,679   | 134        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZEGBHP5JSC73P78H9DEBZ26 | 2024-05-23T00:00:00Z | 2024-06-03T06:00:00Z | 270h0m0s       | -230h0m0s       | 8,197,746   | 24,670,577,521 | 212,552,261 | 135        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZEH794E3K3XDE0HQQ5F95PC | 2024-05-23T00:00:00Z | 2024-06-03T06:00:00Z | 270h0m0s       | -30h0m0s        | 8,197,732   | 2,475,700,402  | 24,901,129  | 135        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZEJV6W7Q2DB4QNDSRMENYZP | 2024-05-23T00:00:00Z | 2024-06-03T06:00:00Z | 270h0m0s       | -               | 8,197,732   | 214,911,680    | 9,604,839   | 135        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZEQ8AJ446NGXY0GT48TA3AP | 2024-05-23T00:00:00Z | 2024-06-03T08:00:00Z | 272h0m0s       | -232h0m0s       | 8,257,083   | 24,849,101,994 | 214,094,084 | 136        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZER10R31NW29ZPA46Q0QZ7E | 2024-05-23T00:00:00Z | 2024-06-03T08:00:00Z | 272h0m0s       | -32h0m0s        | 8,257,069   | 2,493,615,800  | 25,585,742  | 136        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZESN6X13FZK77AG8PS2GV42 | 2024-05-23T00:00:00Z | 2024-06-03T08:00:00Z | 272h0m0s       | -               | 8,257,069   | 215,593,662    | 9,093,680   | 136        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZEY4F46HNDMQ2Q1B1GHZ4Q0 | 2024-05-23T00:00:00Z | 2024-06-03T10:00:00Z | 274h0m0s       | -234h0m0s       | 8,345,547   | 25,028,679,501 | 215,645,396 | 137        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZEZ1CEX1S7THATDJ0AZ3K0A | 2024-05-23T00:00:00Z | 2024-06-03T10:00:00Z | 274h0m0s       | -34h0m0s        | 8,345,533   | 2,511,682,175  | 25,734,650  | 137        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZF0MY7NPS8HD9RT4TYZS7J2 | 2024-05-23T00:00:00Z | 2024-06-03T10:00:00Z | 274h0m0s       | -               | 8,345,533   | 217,169,859    | 9,178,235   | 137        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZF4ZMA5FYBAY7SY3F44B9J8 | 2024-05-23T00:00:00Z | 2024-06-03T12:00:00Z | 276h0m0s       | -236h0m0s       | 8,404,605   | 25,211,087,475 | 217,224,920 | 138        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZF5RHB04840QKHPKN49R62P | 2024-05-23T00:00:00Z | 2024-06-03T12:00:00Z | 276h0m0s       | -36h0m0s        | 8,404,591   | 2,529,990,473  | 25,820,869  | 138        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZF7D4HV2QYDMCC2GA8AFZ9C | 2024-05-23T00:00:00Z | 2024-06-03T12:00:00Z | 276h0m0s       | -               | 8,404,591   | 218,754,145    | 9,235,114   | 138        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZFBVPDEXBHQWF8KQCAJH2WP | 2024-05-23T00:00:00Z | 2024-06-03T14:00:00Z | 278h0m0s       | -238h0m0s       | 8,467,855   | 25,392,403,227 | 218,788,938 | 139        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZFCMHZHFE895R4TQEPJZ6QR | 2024-05-23T00:00:00Z | 2024-06-03T14:00:00Z | 278h0m0s       | -38h0m0s        | 8,467,841   | 2,548,200,044  | 25,897,020  | 139        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZFEA5T52YWCN14YFAKJXBR0 | 2024-05-23T00:00:00Z | 2024-06-03T14:00:00Z | 278h0m0s       | -               | 8,467,841   | 220,331,038    | 9,290,704   | 139        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZFJQJ76NHBCERSSTX28T21T | 2024-05-23T00:00:00Z | 2024-06-03T16:00:00Z | 280h0m0s       | -240h0m0s       | 8,511,359   | 25,574,169,688 | 220,346,882 | 140        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZFKKN94Y2R5B45YFV0KHCC3 | 2024-05-23T00:00:00Z | 2024-06-03T16:00:00Z | 280h0m0s       | -40h0m0s        | 8,511,345   | 2,566,430,897  | 25,970,524  | 140        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZFN9AM3NAN90TD4VP4V78WF | 2024-05-23T00:00:00Z | 2024-06-03T16:00:00Z | 280h0m0s       | -               | 8,511,345   | 221,889,561    | 9,329,777   | 140        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZFSKAV6QT02YQBGMB5SAPQE | 2024-05-23T00:00:00Z | 2024-06-03T18:00:00Z | 282h0m0s       | -242h0m0s       | 8,560,103   | 25,755,960,248 | 221,904,323 | 141        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZFTC6FG0VSXE4DR6TBVD530 | 2024-05-23T00:00:00Z | 2024-06-03T18:00:00Z | 282h0m0s       | -42h0m0s        | 8,560,089   | 2,584,668,190  | 26,627,686  | 141        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZFW2NF4XDA2N38AMGKH74J5 | 2024-05-23T00:00:00Z | 2024-06-03T18:00:00Z | 282h0m0s       | -               | 8,560,089   | 224,333,754    | 10,465,824  | 141        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZG0EQ6TGNPZ5YF9XR6E2DBK | 2024-05-23T00:00:00Z | 2024-06-03T20:00:00Z | 284h0m0s       | -244h0m0s       | 8,605,316   | 25,937,756,385 | 223,464,101 | 142        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZG18E6VXJCCZ5QZWCFPC5YK | 2024-05-23T00:00:00Z | 2024-06-03T20:00:00Z | 284h0m0s       | -44h0m0s        | 8,605,302   | 2,602,901,878  | 26,711,928  | 142        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZG2YPJ3QAB3Q5KT0DVNZ8D3 | 2024-05-23T00:00:00Z | 2024-06-03T20:00:00Z | 284h0m0s       | -               | 8,605,302   | 225,934,526    | 10,535,387  | 142        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZG7AV4JFKFB8RF9ZCRKY2RB | 2024-05-23T00:00:00Z | 2024-06-03T22:00:00Z | 286h0m0s       | -246h0m0s       | 8,647,601   | 26,119,569,497 | 225,021,429 | 143        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZG84PYB6TSZNDM7EPV59G3Q | 2024-05-23T00:00:00Z | 2024-06-03T22:00:00Z | 286h0m0s       | -46h0m0s        | 8,647,587   | 2,621,135,483  | 26,785,036  | 143        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZG9V038C62YPXEWHW2SF59P | 2024-05-23T00:00:00Z | 2024-06-03T22:00:00Z | 286h0m0s       | -               | 8,647,587   | 226,475,580    | 10,578,999  | 143        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZGE67Z24M0QY4DBERYQ1BPM | 2024-05-23T00:00:00Z | 2024-06-04T00:00:00Z | 288h0m0s       | -248h0m0s       | 8,690,564   | 26,301,388,780 | 226,579,538 | 144        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZGEZTWR0AGTY04W4JXGNEKW | 2024-05-23T00:00:00Z | 2024-06-04T00:00:00Z | 288h0m0s       | -48h0m0s        | 8,690,550   | 2,639,371,158  | 26,852,836  | 144        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZGGPS2RQXKPSYSV3TGJB41J | 2024-05-23T00:00:00Z | 2024-06-04T00:00:00Z | 288h0m0s       | -               | 8,690,550   | 229,052,708    | 10,624,943  | 144        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZGN22FA38ZKZAF5KG8GAPE8 | 2024-05-23T00:00:00Z | 2024-06-04T02:00:00Z | 290h0m0s       | -250h0m0s       | 8,734,040   | 26,483,201,383 | 228,138,684 | 145        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZGP4B4ZMNW89EN4DZG1AZWD | 2024-05-23T00:00:00Z | 2024-06-04T02:00:00Z | 290h0m0s       | -50h0m0s        | 8,734,026   | 2,657,610,495  | 26,927,301  | 145        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZGR5F1TBYCV42B0KX3GJK7B | 2024-05-23T00:00:00Z | 2024-06-04T02:00:00Z | 290h0m0s       | -               | 8,734,026   | 230,628,624    | 10,681,828  | 145        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZGVXVB775WCR1MQ0WK4KAS0 | 2024-05-23T00:00:00Z | 2024-06-04T04:00:00Z | 292h0m0s       | -252h0m0s       | 8,775,780   | 26,665,005,994 | 229,696,212 | 146        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZGWQQA2TT2EHNY1FQPMF3SG | 2024-05-23T00:00:00Z | 2024-06-04T04:00:00Z | 292h0m0s       | -52h0m0s        | 8,775,766   | 2,675,841,652  | 26,991,339  | 146        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZGYFKSB2ZCN3DPRS15K51DM | 2024-05-23T00:00:00Z | 2024-06-04T04:00:00Z | 292h0m0s       | -               | 8,775,766   | 231,140,638    | 10,731,788  | 146        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZH2SBSCP5XCRR2HS4DQAGFF | 2024-05-23T00:00:00Z | 2024-06-04T06:00:00Z | 294h0m0s       | -254h0m0s       | 8,821,221   | 26,846,736,757 | 231,257,275 | 147        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZH3KR8R5GWGRZ4RVEYCRKZR | 2024-05-23T00:00:00Z | 2024-06-04T06:00:00Z | 294h0m0s       | -54h0m0s        | 8,821,207   | 2,694,076,877  | 27,667,731  | 147        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZH5CQ6GRW564ZHNAA5F4TKS | 2024-05-23T00:00:00Z | 2024-06-04T06:00:00Z | 294h0m0s       | -               | 8,821,207   | 233,791,134    | 10,750,979  | 147        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZH9NE2MDXTKE1S82D65TCBP | 2024-05-23T00:00:00Z | 2024-06-04T08:00:00Z | 296h0m0s       | -256h0m0s       | 8,882,508   | 27,026,821,693 | 232,816,037 | 148        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZHAFY8N4KDBHYTJB4MMG347 | 2024-05-23T00:00:00Z | 2024-06-04T08:00:00Z | 296h0m0s       | -56h0m0s        | 8,882,494   | 2,712,148,443  | 27,756,355  | 148        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZHC8X9DYNX59ZT5P4MPE8S4 | 2024-05-23T00:00:00Z | 2024-06-04T08:00:00Z | 296h0m0s       | -               | 8,882,494   | 235,337,654    | 10,808,734  | 148        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZHGH9DM31M0AQH8RS4EFB91 | 2024-05-23T00:00:00Z | 2024-06-04T10:00:00Z | 298h0m0s       | -258h0m0s       | 9,027,600   | 27,207,682,330 | 234,451,986 | 149        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZHHDY121E1FMQNWMH9K7SNA | 2024-05-23T00:00:00Z | 2024-06-04T10:00:00Z | 298h0m0s       | -58h0m0s        | 9,027,586   | 2,730,393,737  | 27,923,636  | 149        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZHK7BPC3W2HV2X9C78B76Y5 | 2024-05-23T00:00:00Z | 2024-06-04T10:00:00Z | 298h0m0s       | -               | 9,027,586   | 236,007,768    | 10,954,660  | 149        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZHQDK24EPA8SEGZ2ZM6HZ3G | 2024-05-23T00:00:00Z | 2024-06-04T12:00:00Z | 300h0m0s       | -260h0m0s       | 9,072,566   | 27,389,437,542 | 236,006,996 | 150        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZHR82YXMVWN2CCBQDW7FV7E | 2024-05-23T00:00:00Z | 2024-06-04T12:00:00Z | 300h0m0s       | -60h0m0s        | 9,072,552   | 2,748,622,804  | 27,988,619  | 150        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZHT15GB7AJ0P8PK9NB7CR5Z | 2024-05-23T00:00:00Z | 2024-06-04T12:00:00Z | 300h0m0s       | -               | 9,072,552   | 238,560,617    | 11,007,786  | 150        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZHY8R8A17X6494SZGR7CXAV | 2024-05-23T00:00:00Z | 2024-06-04T14:00:00Z | 302h0m0s       | -262h0m0s       | 9,121,538   | 27,570,451,661 | 237,561,081 | 151        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZHZ37Z9WDJ6GMZ2VTEXXK3T | 2024-05-23T00:00:00Z | 2024-06-04T14:00:00Z | 302h0m0s       | -62h0m0s        | 9,121,524   | 2,766,790,176  | 28,062,111  | 151        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZJ0WSN5YYHK7773AKHGTWGQ | 2024-05-23T00:00:00Z | 2024-06-04T14:00:00Z | 302h0m0s       | -               | 9,121,524   | 240,134,797    | 11,060,610  | 151        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZJ5488P22RR7RD5NDY23YDK | 2024-05-23T00:00:00Z | 2024-06-04T16:00:00Z | 304h0m0s       | -264h0m0s       | 9,162,598   | 27,751,143,228 | 239,079,518 | 152        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZJ5YWD69741HWWPNC3WN97E | 2024-05-23T00:00:00Z | 2024-06-04T16:00:00Z | 304h0m0s       | -64h0m0s        | 9,162,584   | 2,784,910,200  | 28,125,794  | 152        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZJ7RY9N6WA0TCCGW7C363H8 | 2024-05-23T00:00:00Z | 2024-06-04T16:00:00Z | 304h0m0s       | -               | 9,162,584   | 240,692,643    | 11,102,549  | 152        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 1h0m0s     | compactor |
| 01HZJC00EGS3BWJ3KNJ9PDDZT4 | 2024-05-23T00:00:00Z | 2024-06-04T18:00:00Z | 306h0m0s       | -266h0m0s       | 9,201,482   | 27,931,865,789 | 240,625,169 | 153        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01HZJCWJZPS1GFP4SPAJE7P9BJ | 2024-05-23T00:00:00Z | 2024-06-04T18:00:00Z | 306h0m0s       | -66h0m0s        | 9,201,468   | 2,803,031,682  | 28,766,076  | 153        | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 5m0s       | compactor |
| 01HZDC10JAG5YQ5K77TT243EM1 | 2024-06-02T18:00:00Z | 2024-06-02T20:00:00Z | 1h59m59.998s   | 38h0m0.002s     | 793,096     | 180,068,219    | 1,541,972   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZDJWQSZKGJBYSF2EQKCK9BS | 2024-06-02T20:00:00Z | 2024-06-02T22:00:00Z | 1h59m59.995s   | 38h0m0.005s     | 846,928     | 176,417,999    | 1,555,779   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZDSRF2859R96AH4VNH84X4T | 2024-06-02T22:00:00Z | 2024-06-03T00:00:00Z | 1h59m59.998s   | 38h0m0.002s     | 777,905     | 175,701,963    | 1,507,267   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZE0M69Z2XD18Q0FT5ZSM35Z | 2024-06-03T00:00:00Z | 2024-06-03T02:00:00Z | 1h59m59.988s   | 38h0m0.012s     | 778,487     | 176,150,089    | 1,510,278   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZE7FXJ49GEC0824A8TZHC9Z | 2024-06-03T02:00:00Z | 2024-06-03T04:00:00Z | 1h59m59.999s   | 38h0m0.001s     | 802,718     | 175,933,623    | 1,529,912   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZEEBMSY06C67WBFSZ6QVB9R | 2024-06-03T04:00:00Z | 2024-06-03T06:00:00Z | 1h59m59.996s   | 38h0m0.004s     | 795,159     | 178,845,928    | 1,535,758   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZEN7C7WC8R6BEPZ0ZKFEY21 | 2024-06-03T06:00:00Z | 2024-06-03T08:00:00Z | 1h59m59.997s   | 38h0m0.003s     | 813,335     | 178,524,473    | 1,541,823   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZEW33HKX4Y8MZ34GJ78PEZN | 2024-06-03T08:00:00Z | 2024-06-03T10:00:00Z | 2h0m0s         | 38h0m0s         | 833,428     | 179,577,507    | 1,551,312   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZF2YTQFENGVRG5MJ7AZ294T | 2024-06-03T10:00:00Z | 2024-06-03T12:00:00Z | 1h59m59.999s   | 38h0m0.001s     | 823,896     | 182,407,974    | 1,579,524   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZF9THZS9P1DJHT00C080Z9R | 2024-06-03T12:00:00Z | 2024-06-03T14:00:00Z | 1h59m59.999s   | 38h0m0.001s     | 825,754     | 181,315,752    | 1,564,018   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZFGP98CS5NYWD5QK0E9V3D0 | 2024-06-03T14:00:00Z | 2024-06-03T16:00:00Z | 1h59m59.999s   | 38h0m0.001s     | 802,463     | 181,766,461    | 1,557,944   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZFQJ0NKQN83EJ5BQ7VNA2M6 | 2024-06-03T16:00:00Z | 2024-06-03T18:00:00Z | 1h59m59.989s   | 38h0m0.011s     | 808,165     | 181,790,560    | 1,557,441   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZFYDQPXDAEM03EMKHE8GK9B | 2024-06-03T18:00:00Z | 2024-06-03T20:00:00Z | 1h59m59.995s   | 38h0m0.005s     | 804,467     | 181,796,137    | 1,559,778   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZG59EZTDKRRNDN0XDCG8PX0 | 2024-06-03T20:00:00Z | 2024-06-03T22:00:00Z | 1h59m59.992s   | 38h0m0.008s     | 801,650     | 181,813,112    | 1,557,328   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZGC567ZEV3KAJP7GT5V65YM | 2024-06-03T22:00:00Z | 2024-06-04T00:00:00Z | 1h59m59.99s    | 38h0m0.01s      | 802,417     | 181,819,283    | 1,558,109   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZGK0XG9ZEWVTRG8TAYHAZZX | 2024-06-04T00:00:00Z | 2024-06-04T02:00:00Z | 1h59m59.995s   | 38h0m0.005s     | 803,038     | 181,812,603    | 1,559,146   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZGSWMJ7XCAJQX3605E9N0ZD | 2024-06-04T02:00:00Z | 2024-06-04T04:00:00Z | 1h59m59.997s   | 38h0m0.003s     | 801,373     | 181,804,611    | 1,557,528   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZH0RBSY79J2MQH7ZT2G97EA | 2024-06-04T04:00:00Z | 2024-06-04T06:00:00Z | 1h59m59.998s   | 38h0m0.002s     | 804,927     | 181,730,763    | 1,561,063   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZH7M31PK1E4C8RRXHKZP13C | 2024-06-04T06:00:00Z | 2024-06-04T08:00:00Z | 1h59m59.992s   | 38h0m0.008s     | 813,400     | 180,084,936    | 1,558,762   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZHEFT9Z7REC7ANXF2WDWSRC | 2024-06-04T08:00:00Z | 2024-06-04T10:00:00Z | 2h0m0s         | 38h0m0s         | 901,725     | 180,860,637    | 1,635,949   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZHNBHHQAX0MWQXRP2HZZXCT | 2024-06-04T10:00:00Z | 2024-06-04T12:00:00Z | 1h59m59.997s   | 38h0m0.003s     | 803,414     | 181,755,212    | 1,555,010   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZHW78YV9YEABR5MEQN3FEYA | 2024-06-04T12:00:00Z | 2024-06-04T14:00:00Z | 1h59m59.999s   | 38h0m0.001s     | 808,864     | 181,014,119    | 1,554,085   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZJ3309R74REVKK2FVHV4T7E | 2024-06-04T14:00:00Z | 2024-06-04T16:00:00Z | 1h59m59.996s   | 38h0m0.004s     | 796,151     | 180,691,567    | 1,518,437   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HZJ9YQG0J3FXCQ4B5HN5DKF4 | 2024-06-04T16:00:00Z | 2024-06-04T18:00:00Z | 1h59m59.978s   | 38h0m0.022s     | 793,762     | 180,722,561    | 1,545,651   | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01HKS51EDP1BNCCP4FX49F87T1 | 2033-01-02T18:00:00Z | 2033-01-03T00:00:00Z | 5h59m59.449s   | 34h0m0.551s     | 8,761       | 6,306,500      | 52,556      | 2          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | compactor |
| 01SVRZ26G0NEE0W72ZFE6860FV | 2033-01-03T00:00:00Z | 2033-01-03T02:00:00Z | 1h59m59.449s   | 38h0m0.551s     | 8,759       | 2,102,160      | 17,518      | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
| 01SVS5XXQT99Z6JHT78PDSVE2Z | 2033-01-03T02:00:00Z | 2033-01-03T04:00:00Z | 1h59m59.449s   | 38h0m0.551s     | 8,761       | 2,102,180      | 17,520      | 1          | false       | cluster=blr-ocp2,prometheus=dev-ops-tools/prometheus-ha-kube-prometh-prometheus,prometheus_replica=prometheus-prometheus-ha-kube-prometh-prometheus-1 | 0s         | sidecar   |
level=info ts=2024-06-04T20:32:02.547360503Z caller=main.go:164 msg=exiting

meta.json:
meta (11).json

Thanos Bucket View/Screenshot:

Environment:

• Running in Openshift 4.12.53
• Kubernetes version v1.25.16+9946c63

[GiedriusS]

That's such an old version. Is it the same on the newest one?

[jbarber-rbbn]

@GiedriusS , all our clusters are running the same version.
I can't say if this is or isn't an issue in a newer version.

We have 8 clusters with 2 prometheus deployments per cluster and this is the only deployment* with the issue.

[douglascamata]

Compaction level can go beyond 4, it's normal. Particularly I see it happening a lot when there are restarts in Thanos Receive. Are you using Receive or Sidecar?