read label value index: invalid checksum

[tinytub]

What did you do?
get all labels by api /api/v1/label/__name__/values and got error

I use 6 server to make a cluster
1 center server use to proxy all query request to backend server
others do scrape jobs

my prome02,prome03 server got the invalid checksum error

prome02 sample append

prome03 sample append

the high light line is the error folder.

What did you expect to see?
get all labels with no error

What did you see instead? Under which circumstances?

{
  "status": "error",
  "errorType": "execution",
  "error": "block: 01C4HGBJWVNH2HACQ8T941R3YV: read label value index: invalid checksum"
}

Environment

• System information:

Linux 2.6.32-696.3.2.el6.v6.2.x86_64 x86_64

• Prometheus version:

prometheus, version 2.1.0 (branch: HEAD, revision: 85f23d8)
build user: root@6e784304d3ff
build date: 20180119-12:01:23
go version: go1.9.2

• Prometheus configuration file:

# center server only do remote_read

remote_read:
  - url: "http://prome01:9090/api/v1/read"
    remote_timeout: 300s
  - url: "http://prome02:9090/api/v1/read"
    remote_timeout: 300s
  - url: "http://prome03:9090/api/v1/read"
    remote_timeout: 300s
  - url: "http://prome04:9090/api/v1/read"
    remote_timeout: 300s
  - url: "http://prome05:9090/api/v1/read"
    remote_timeout: 300s

#backend server do scrape job,and use hashmod.

scrape_configs:
  - job_name: 'wonder-transfer'
    scrape_interval: 30s
    scrape_timeout: 30s
    static_configs:
       - targets:
          - 'k12735v:6061'
          ...
          ...
          - 'k4645v:6061'
    metric_relabel_configs:
    - source_labels: [bid]
      modulus:       5 
      target_label:  __tmp_hash
      action:        hashmod
    - source_labels: [__tmp_hash]
      regex:         ^4$  
      action:        keep
    - target_label: instance
      replacement: Transfer

[Techcadia]

@tinytub Can I ask how you solved this problem. I seem to be having the same issue. Currently running on 2.1.0 as well.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.