The receiver logs content massive waring logs as well as promethues side 409 conflict error

[JayChanggithub]

Hi team,

We found weird symptoms that we using thanos-query to retrieve how much prometheus has been in thanos-receiver. Also the counts of value whether should be stable(The screenshot as below)? Since we wonder know how much prometheus already retrieve. Actually we has been adopt around 23+ clusters. However it's only showing 4 clusters. it's very weird.

In additional. I get into several promethues which setting remote_write to thanos-receiver have massive error logs as below:

ts=2023-02-24T03:15:02.269Z caller=dedupe.go:112 component=remote level=error remote_name=ea11e1 url=http://thanos-receive.thanos.kyma.ondemand.com/api/v1/receive msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 409 Conflict: 3 errors: forwarding request to endpoint thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = store locally for endpoint thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901: add 1 samples: out of order sample; forwarding request to endpoint thanos-receive-3.thanos-receive.thanos.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = store locally for endpoint thanos-receive-3.thanos-receive.thanos.svc.cluster.local:10901: add 1 samples: out of order sample; store locally for endpoint thanos-receive-4.thanos-receive.thanos.svc.cluster.local:10901: add 1 samples: out of order sample"

Meantime i also get into thanos-receiver to view logs as below:

level=warn ts=2023-02-24T03:20:39.078321844Z caller=writer.go:188 component=receive component=receive-writer tenant=default-tenant msg="Error on ingesting samples that are too old or are too far into the future" numDropped=92

more detailed information

• promethus version: app.kubernetes.io/version: 2.40.5

• thanos version: thanosio/thanos:v0.30.2

• thanos-receiver args:

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: thanos-receive-hashrings
    namespace: thanos
  data:
    thanos-receive-hashrings.json: |
      [
        {
          "hashring": "soft-tenants",
          "endpoints":
          [
              "thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901",
              "thanos-receive-1.thanos-receive.thanos.svc.cluster.local:10901",
              "thanos-receive-2.thanos-receive.thanos.svc.cluster.local:10901",
              "thanos-receive-3.thanos-receive.thanos.svc.cluster.local:10901",
              "thanos-receive-4.thanos-receive.thanos.svc.cluster.local:10901"
          ]
        }
      ]
  ---

  spec:
    replicas: 5
    selector:
      matchLabels:
        kubernetes.io/name: thanos-receive
    serviceName: thanos-receive
    template:
      metadata:
        labels:
          kubernetes.io/name: thanos-receive
          thanos-store-api: "true"
      spec:
        containers:
        - args:
          - receive
          - --grpc-address=0.0.0.0:10901
          - --http-address=0.0.0.0:10902
          - --remote-write.address=0.0.0.0:19291
          - --receive.replication-factor=3
          - --receive.hashrings-algorithm=ketama
          - --objstore.config-file=/etc/thanos/objectstorage.yaml
          - --tsdb.path=/var/thanos/receive
          - --tsdb.retention=12h
          - --receive-forward-timeout=120s
          - --label=receive_replica="$(NAME)"
          - --label=receive="true"
          - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json
          - --receive.local-endpoint=$(NAME).thanos-receive.thanos.svc.cluster.local:10901