compaction eating up disk space

[ftaffelt]

What did you do?
clean 2.2 installation. (had troubles because of #3956)

What did you see instead? Under which circumstances?
after running normally for about a day, all free disk space is used up during compacting.

Environment

• System information:

  Windows

• Prometheus version:

  prometheus, version 2.2.0 (branch: HEAD, revision: f63e7db)
  build user: root@52af9f66ce71
  build date: 20180308-16:53:11
  go version: go1.10

• Logs:

level=error ts=2018-03-14T03:27:03.480912Z caller=db.go:281 component=tsdb msg="compaction failed" err="reload blocks: invalid block sequence: block time ranges overlap (1520856000000, 1520920800000)"
level=info ts=2018-03-14T03:30:03.6236144Z caller=compact.go:394 component=tsdb msg="compact blocks" count=12 mint=1520920800000 maxt=1520985600000
level=error ts=2018-03-14T03:30:57.2157271Z caller=compact.go:404 component=tsdb msg="removed tmp folder after failed compaction" err="remove e:\\prometheus\\prometheus\\data\\01C8H8XG75ZNS506TDS7BEBN9E.tmp\\chunks\\000011: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird."
level=error ts=2018-03-14T03:30:57.9270709Z caller=db.go:281 component=tsdb msg="compaction failed" err="compact [e:\\prometheus\\prometheus\\data\\01C8H7ANP6309XEWJ2QFA9RM41 e:\\prometheus\\prometheus\\data\\01C8H76G120CZ3N4M16QVVEBDS e:\\prometheus\\prometheus\\data\\01C8H7Z5GRW0DK65A5S4640VRH e:\\prometheus\\prometheus\\data\\01C8H76K6HHP8R7PA6EJSPGMA3 e:\\prometheus\\prometheus\\data\\01C8H78JGV29KCREAK59AQCQRS e:\\prometheus\\prometheus\\data\\01C8H855F1M5B18GR5VGBW75KW e:\\prometheus\\prometheus\\data\\01C8H7D16X6N8XHZKSFP8GHGB5 e:\\prometheus\\prometheus\\data\\01C8H7FX048Z5002BV5JRA23SD e:\\prometheus\\prometheus\\data\\01C8H7KS63JZ3EPHG6TSAA0QXY e:\\prometheus\\prometheus\\data\\01C8H7SDV2FMCKS5N6YAF109EJ e:\\prometheus\\prometheus\\data\\01C8H8BZHDBVTH2BRHN7QY47XG e:\\prometheus\\prometheus\\data\\01C8H76FM0B7AX4X5NW5NSC3JJ]: write compaction: write chunks: truncate e:\\prometheus\\prometheus\\data\\01C8H8XG75ZNS506TDS7BEBN9E.tmp\\chunks\\000011: Es steht nicht genug Speicherplatz auf dem Datenträger zur Verfügung."
level=info ts=2018-03-14T05:00:00.5051057Z caller=compact.go:394 component=tsdb msg="compact blocks" count=1 mint=1520992800000 maxt=1521000000000
level=error ts=2018-03-14T05:00:01.1722885Z caller=db.go:281 component=tsdb msg="compaction failed" err="reload blocks: invalid block sequence: block time ranges overlap (1520856000000, 1520985600000)" 

[n0guest]

It could be related to #3943. At least i got the same problem #3943 (comment).

[trnl]

@ftaffelt , any chance to reproduce it with 2.2.1?

[ftaffelt]

updated my installation to 2.2.1 . looks good so far.

[ftaffelt]

as the issue never occurred after upgrading to 2.2.1, shall i close this issue ?

[colonha]

Still happening in version 2.2.1. with fresh data.

level=error ts=2018-04-25T15:34:18.059820099Z caller=db.go:281 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set

[shaoxt]

Still saw this issue in version 2.4.3. My prometheus is running on K8S.
the reason is that the prometheus might be killed by K8S due to OOM. It left "*.tmp" in the disk.
No one is going to clean it up. In prometheus, it should have a something to cleanup the dead tmp directory.