compact error leading to disk entire in a hour in 2.2.1

[zhouyuanchao]

#3714
this issue shows this bug is fix in 2.2.1, but when I use 2.2.1, disk was entire in a hour.

level=error ts=2018-05-16T06:11:11.282000739Z caller=db.go:281 component=tsdb msg="compaction failed" err="compact [/opt/huawei/prometheus/data/01CDFJQJA3BKVWCJD0YKZ0XABV /opt/huawei/prometheus/data/01CDHGH6P0AQY2KK36B93WM54W /opt/huawei/prometheus/data/01CDKEB3S50X0DKHBJXAFP46B0]: mmap files: mmap: cannot allocate memory"
level=error ts=2018-05-16T06:14:11.283045419Z caller=db.go:281 component=tsdb msg="compaction failed" err="compact [/opt/huawei/prometheus/data/01CDFJQJA3BKVWCJD0YKZ0XABV /opt/huawei/prometheus/data/01CDHGH6P0AQY2KK36B93WM54W /opt/huawei/prometheus/data/01CDKEB3S50X0DKHBJXAFP46B0]: mmap files: mmap: cannot allocate memory"

level=info ts=2018-05-16T10:34:51.201694208Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1526443200000 maxt=1526450400000
level=error ts=2018-05-16T10:35:21.361394055Z caller=db.go:281 component=tsdb msg="compaction failed" err="reload blocks: open block /opt/huawei/prometheus/data/01CDKVZHD33HJQKXWV3DS955M5: mmap files: mmap: cannot allocate memory"

when i try to restart
level=info ts=2018-05-16T11:34:39.020138102Z caller=main.go:407 msg="Notify discovery manager stopped"
level=error ts=2018-05-16T11:34:39.020250764Z caller=main.go:582 err="Opening storage failed open block /opt/huawei/prometheus/data/01CDM66MBXFR2KS3X63Z4REXMH: mmap files: mmap: cannot allocate memory"
level=info ts=2018-05-16T11:34:39.020290097Z caller=main.go:584 msg="See you next time!"

[krasi-georgiev]

mmap: cannot allocate memory you need more memory 😄

[zhouyuanchao]

I still have 30G free memory

[krasi-georgiev]

this error comes from the OS , maybe some limit on what Prometheus can use, but in any way this doesn't look like a bug.

[krasi-georgiev]

if you run a search all issues with cannot allocate memory you will see that every report was due to insufficient system memory.

[zhouyuanchao]

The meta.json in all dir is same, why write to disk repeatly?

[krasi-georgiev]

find the culprit for the memory error(not enough physical memory or some memory restrictions) and open a new issue if you still find that there is a problem.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.