Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upprometheus data loss with error "Error indexing label pair to fingerprints batch...." #2812
Comments
garyyang85
closed this
Jun 7, 2017
This comment has been minimized.
This comment has been minimized.
|
seems due to ssd unstable, after change to another env without ssd, it works |
This comment has been minimized.
This comment has been minimized.
lock
bot
commented
Mar 23, 2019
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
lock
bot
locked and limited conversation to collaborators
Mar 23, 2019
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
garyyang85 commentedJun 6, 2017
I am using kubernetes + prometheus + node exporter for node cpu/memory/disk monitor. Now the prometheus work very unstable. I have 3 node in kubernetes, at first it lost cpu data on node1 and node2, after restart it lost cpu data on node1 and disk data on node3. I found below errors in logs:
time="2017-06-06T11:03:39Z" level=warning msg="Flag -storage.local.memory-chunks is deprecated. Its value 2097152 is used to override -storage.local.target-heap-size to 6442450944." source="config.go:317"
time="2017-06-06T11:03:39Z" level=info msg="Starting prometheus (version=1.6.3, branch=master, revision=c580b60c67f2c5f6b638c3322161bcdf6d68d7fc)" source="main.go:88"
time="2017-06-06T11:03:39Z" level=info msg="Build context (go=go1.8.1, user=root@a6410e65f5c7, date=20170522-09:15:06)" source="main.go:89"
time="2017-06-06T11:03:39Z" level=info msg="Loading configuration file /prometheus/prometheus-sailoros.yml" source="main.go:251"
time="2017-06-06T11:03:39Z" level=info msg="Loading series map and head chunks..." source="storage.go:421"
time="2017-06-06T11:03:39Z" level=info msg="0 series loaded." source="storage.go:432"
time="2017-06-06T11:03:39Z" level=info msg="Starting target manager..." source="targetmanager.go:61"
time="2017-06-06T11:03:39Z" level=info msg="Listening on :9090" source="web.go:259"
time="2017-06-06T11:03:39Z" level=info msg="Using pod service account via in-cluster config" source="kubernetes.go:94"
time="2017-06-06T11:03:39Z" level=info msg="Using pod service account via in-cluster config" source="kubernetes.go:94"
time="2017-06-06T11:03:51Z" level=error msg="Error indexing label pair to fingerprints batch: open data/labelpair_to_fingerprints/000002.log: no such file or directory" source="persistence.go:1410"
time="2017-06-06T11:03:51Z" level=error msg="The storage is now inconsistent. Restart Prometheus ASAP to initiate recovery." error="open data/labelpair_to_fingerprints/000002.log: no such file or directory" source="persistence.go:415"
time="2017-06-06T11:03:59Z" level=error msg="Error indexing label pair to fingerprints batch: open data/labelpair_to_fingerprints/000002.log: no such file or directory" source="persistence.go:1410"
time="2017-06-06T11:08:39Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-06-06T11:08:39Z" level=error msg="Error while checkpointing: open data/heads.db.tmp: no such file or directory" source="storage.go:1468"
time="2017-06-06T11:13:39Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-06-06T11:13:39Z" level=error msg="Error while checkpointing: open data/heads.db.tmp: no such file or directory" source="storage.go:1468"
time="2017-06-06T11:18:39Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-06-06T11:18:39Z" level=error msg="Error while checkpointing: open data/heads.db.tmp: no such file or directory" source="storage.go:1468"
Could you help about this? Thanks you very much!