Large chunks

[matejkramny]

What did you do?

nothing / n/a

What did you expect to see?

Reasonable usage of disk space

What did you see instead? Under which circumstances?

Disk space usage by prometheus grew substantially.

% du -ha / 2>/dev/null | sort -r --human-numeric-sort | head -n 32 

221G    /
214G    /opt/docker/metrics/_data/prometheus
214G    /opt/docker/metrics/_data
214G    /opt/docker/metrics
214G    /opt/docker
214G    /opt
81G    /opt/docker/metrics/_data/prometheus/01C92PFPNFBCXR0YZMK2H8ANG8
78G    /opt/docker/metrics/_data/prometheus/01C92PFPNFBCXR0YZMK2H8ANG8/chunks
50G    /opt/docker/metrics/_data/prometheus/01C92NXXAJ6TM51ZT1VBCM43KZ
48G    /opt/docker/metrics/_data/prometheus/01C92NXXAJ6TM51ZT1VBCM43KZ/chunks
31G    /opt/docker/metrics/_data/prometheus/01C92NH37S61MQ9FSRGMZ6B2MD
30G    /opt/docker/metrics/_data/prometheus/01C92NH37S61MQ9FSRGMZ6B2MD/chunks
20G    /opt/docker/metrics/_data/prometheus/01C92N6CYMX3JG772S0YD7604E
19G    /opt/docker/metrics/_data/prometheus/01C92N6CYMX3JG772S0YD7604E/chunks
12G    /opt/docker/metrics/_data/prometheus/01C92MY6FF5WRY8VSS5TCN0FQ2/chunks
12G    /opt/docker/metrics/_data/prometheus/01C92MY6FF5WRY8VSS5TCN0FQ2
7.3G    /opt/docker/metrics/_data/prometheus/01C92MQ4C7ZWN976PJ2J1EDH79
7.0G    /opt/docker/metrics/_data/prometheus/01C92MQ4C7ZWN976PJ2J1EDH79/chunks
4.6G    /opt/docker/metrics/_data/prometheus/01C92MGQ8BPTKCA6HC1J59DC47
4.5G    /var
4.4G    /opt/docker/metrics/_data/prometheus/01C92MGQ8BPTKCA6HC1J59DC47/chunks
4.1G    /var/lib
3.9G    /var/lib/prometheus
3.6G    /opt/docker/metrics/_data/prometheus/01C92PFPNFBCXR0YZMK2H8ANG8/index
2.8G    /opt/docker/metrics/_data/prometheus/01C92MAQWZ4JTJEQSRRY58Q94V
2.7G    /opt/docker/metrics/_data/prometheus/01C92MAQWZ4JTJEQSRRY58Q94V/chunks
2.2G    /opt/docker/metrics/_data/prometheus/01C92NXXAJ6TM51ZT1VBCM43KZ/index
1.8G    /opt/docker/metrics/_data/prometheus/01C92M4WPRN7Q68H3PZCZ25ND1
1.7G    /usr
1.7G    /opt/docker/metrics/_data/prometheus/01C92M4WPRN7Q68H3PZCZ25ND1/chunks
1.4G    /opt/docker/metrics/_data/prometheus/01C92NH37S61MQ9FSRGMZ6B2MD/index
1.1G    /opt/docker/metrics/_data/prometheus/01C92KZ6CNB8CBWE7BTCCTJYRE/chunks

Environment

Docker, prometheus v2.2.0

• System information:

  insert output of uname -srm here

Linux 4.13.0-37-generic x86_64

• Prometheus version:

  insert output of prometheus --version here

2.2.0

• Prometheus configuration file:

insert configuration here

• Logs:

Logs don't show any relevant information. I found a maybe-related issue #3190 however there are no .tmp files, and i don't see any "compaction failed" in the logs.

The logs are normal, aside from occasional scrape failures.

Can something be done about the size of the chunks? It seems unreasonable to me to grow like that (see graph)

[gouthamve]

Hi could you upgrade to 2.2.1 and let us know if that changes anything? I am afraid this might be an issue with 2.2.0

[matejkramny]

Hi, I did already but the data still there. Will I need to delete them or is prometheus able to purge this?

[gouthamve]

You will need to delete them.

[matejkramny]

I will reopen if the issue comes back. Thanks

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.