thanos receiver demo - setup problems - ... not solved

[enimath]

Hello everyone !

I want to build Demo of thanos with receiver setup and I am stuck.

My whish setup in attached draw :

I was trying to acheive this with 3 thanos receiver based on mounted network storage with thanos query and prometheus writing data into receiver.

each receiver/query component = each virtual machine based on debian10 with mounted network storage

thanos version:

thanos, version 0.27.0 (branch: HEAD, revision: f68bb366d18557a5347e6b38013f363ad5be05a4)
  build user:       root@708554d4fa46
  build date:       20220705-14:57:42
  go version:       go1.17.11
  platform:         linux/amd64

prometheus version:

prometheus --version
prometheus, version 2.37.0 (branch: HEAD, revision: b41e0750abf5cc18d8233161560731de05199330)
  build user:       root@0ebb6827e27f
  build date:       20220714-15:13:18
  go version:       go1.18.4
  platform:         linux/amd64

prometheus configuration (I started with 1 prometeus writing to check is this working) :
cat /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP
ExecStart=/usr/local/bin/prometheus  --web.enable-admin-api  \
 --web.enable-lifecycle \
 --storage.tsdb.min-block-duration=2h \
 --storage.tsdb.max-block-duration=2h \
 --config.file=/etc/prometheus/prometheus.yml \
 --storage.tsdb.path=/var/lib/prometheus \
 --web.console.templates=/etc/prometheus/consoles \
 --web.console.libraries=/etc/prometheus/console_libraries \
 --web.listen-address=0.0.0.0:9090 \
 --web.external-url=
SyslogIdentifier=prometheus
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target`

cat /etc/prometheus/prometheus.yml
(( I started without loadbalancers to check if this is working so I setup directly thanos receiver no1

remote_write:
- url: http://10.128.5.131:10908/api/v1/receive
  headers:
   THANOS-TENANT: foo

    # my global config
    global:
      scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      external_labels:
        cluster: thanos

      # scrape_timeout is set to the global default (10s).
    # Alertmanager configuration
    alerting:
      alertmanagers:
        - static_configs:
            - targets:
              # - alertmanager:9093

    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"

    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: "prometheus"

        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.

        static_configs:
          - targets: ["localhost:9090"]
remote_write:
- url: http://10.128.5.130:10908/api/v1/receive
  headers:
   THANOS-TENANT: foo

each thanos receive component :
they are different by " --label "receive_replica="2"" " for example ip.131=1 , ip.132=2 , ip.133=3
I was testing also few options with "receive.replication-factor" with 1 , 2 , 3 ( I ended with 2)
10.128.5.131 , 10.128.5.132 , 10.128.5.133
also each IP have own folder 131 - thanos1, 132 - thanos2 , 133 - thanos3

cat /etc/systemd/system/thanosreciver.service
[Unit]
Description=ThanosReceiver1
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=thanos
Group=thanos
ExecReload=/bin/kill -HUP
ExecStart=/usr/local/bin/thanos receive \
--tsdb.path "/mnt/moosefs/thanos1" \
--grpc-address 10.128.5.131:10907 \
--receive.local-endpoint 10.128.5.131:10907 \
--http-address 10.128.5.131:10909 \
--remote-write.address 10.128.5.131:10908 \
--log.level=debug \
--log.format=logfmt \
--receive.replication-factor 2 \
--label "receive_replica=\"1\"" \
--label "receive_cluster=\"thanos\"" \
--receive.tenant-label-name=foo \
--receive.hashrings-file /etc/thanos/hashring.json \
--tsdb.max-exemplars=1 \
--objstore.config-file "/etc/thanos/object-store.yaml"

SyslogIdentifier=thanosreceiver1
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target`

cat /etc/thanos/object-store.yaml

    type: FILESYSTEM
    config:
      directory: "/mnt/moosefs/thanos"
    prefix: ""

cat /etc/thanos/hashring.json

[
    {
        "hashring": "foo",
        "endpoints": [
            "10.128.5.131:10907",
            "10.128.5.132:10907",
            "10.128.5.133:10907"
        ],
        "tenants": ["foo"]
    }
]

thanos query setup (also without any load balancer to start it running)

10.128.5.129

cat /etc/systemd/system/thanosquery.service
[Unit]
Description=ThanosQuery
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=thanos
Group=thanos
ExecReload=/bin/kill -HUP
ExecStart=/usr/local/bin/thanos query \
    --log.level=debug \
    --log.format=logfmt \
    --http-address 0.0.0.0:19192 \
    --store 10.128.5.130:19090 \
    --store 10.128.5.131:10907 \
    --store 10.128.5.132:10907 \
    --store 10.128.5.133:10907 \
    --query.replica-label "replica" \
    --query.replica-label "receive_replica" \
    --query.replica-label "region"


SyslogIdentifier=thanosquery
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target

I couldn't start storage next to query cause of "port in use" so I start another VM with storage :
10.128.5.130

cat /etc/systemd/system/thanosstorage.service

[Unit]
Description=ThanosStorage
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=thanos
Group=thanos
ExecReload=/bin/kill -HUP
ExecStart=/usr/local/bin/thanos  store \
    --data-dir /var/thanos/store \
    --objstore.config-file /etc/thanos/object-store.yaml \
    --http-address         0.0.0.0:10901 \
    --grpc-address         0.0.0.0:19090

SyslogIdentifier=thanosstorage
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target

cat /etc/thanos/object-store.yaml

    type: FILESYSTEM
    config:
      directory: "/mnt/moosefs/thanos"
    prefix: ""

when I start everything there are errors ... I was changing settings so many times to run it so I don't remember what configurations I have already tested... 

I had few errors :

prometheus component=remote level=warn remote_name=ca1822 url=http://10.128.5.131:10908/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 500

and

thanos level=error caller=proxy.go:70 err transport: the stream is done or WriteHeader was already called 

and

prometheus[1975006]: ts=2022-07-18T10:18:33.858Z caller=dedupe.go:112 component=remote level=warn remote_name=ca1822 url=http://10.128.5.131:10908/api/v1/receive msg="Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: context deadline exceeded"

UPDATE !

when 1 receiver goes down ... there is no scrapping..

[enimath]

now I am wondering should I use thanos storage or not ?

another problem occured during this instalation is that when I shut down one receiver...
i cannot write from prometheus side error 500

msg="Failed to send batch, retrying" err="server returned HTTP status 500 Internal Server Error: context deadline exceeded"

from receivers side :

`Jul 22 12:15:01 thanos-02 thanosreceiver2[882]: level=debug ts=2022-07-22T12:15:01.783548281Z caller=handler.go:414 component=receive component=receive-handler tenant=foo msg="failed to handle request" err="context deadline exceeded"
Jul 22 12:15:01 thanos-02 thanosreceiver2[882]: level=error ts=2022-07-22T12:15:01.783671918Z caller=handler.go:429 component=receive component=receive-handler tenant=foo err="context deadline exceeded" msg="internal server error"
Jul 22 12:15:01 thanos-02 thanosreceiver2[882]: level=debug ts=2022-07-22T12:15:01.784350687Z caller=handler.go:650 component=receive component=receive-handler msg="request failed, but not needed to achieve quorum" err="forwarding request to endpoint 10.128.5.131:10907: rpc error: code = DeadlineExceeded desc = context deadline exceeded"
`

[enimath]

I just remove from prometheus extra external labels and also from query and deleted all data with this deduplicate labels.
and I dont know how ... now it is working ...
and I can shut down one receiver and is running

also I used 2x load balancer VM on receiver side and 2x loadbalancers on query side each balancer have haproxy and keepalived.
Here is example of grafana reading from load balancer data scrapped from proxmox prometheus :

[enimath]

I am abit frustrated... now problem again occure.. when I stop 1 of 3 receivers (replica 2x) scrapping is stopped due to :
"Error on ingesting out-of-order samples"

and

Aug 12 09:53:11 hv-03 prometheus[1826385]: ts=2022-08-12T09:53:11.374Z caller=dedupe.go:112 component=remote level=error remote_name=a8d36d url=http://10.128.5.110:10908/api/v1/receive msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 409 Conflict: 3 errors: replicate write request for endpoint 10.128.5.132:10907: quorum not reached: 2 errors: backing off forward request for endpoint 10.128.5.133:10907: target not available; store locally for endpoint 10.128.5.132:10907: conflict; replicate write request for endpoint 10.128.5.133:10907: quorum not reached: 2 errors: backing off forward request for endpoint 10.128.5.133:10907: target not available; forwarding request to endpoint 10.128.5.131:10907: rpc error: code = AlreadyExists desc = store locally for endpoint 10.128.5.131:10907: conflict; replicate write request for endpoint 10.128.5.131:10907: quorum not reached: conflict"

anyone could help to debug this issue ?

[apoorva-marisomaradhya]

Hi, are you still having the issue?
We do have similar setup, 5 receivers and replication-factor set to 3 and see the below error in receiver logs
msg="Error on ingesting samples with different value but same timestamp" numDropped=1

Any help will be much appreciated!

[enimath]

Hello,

I just made setup with 5 receivers not 3 with replica 2x but in my opinion this is something that need to be fixed.