keep on getting "disk quota exceeded" Log on wal file

[TroubleshooteRJ]

hello team,

I am running prometheus, version 2.3.1 (branch: HEAD, revision: 188ca45)

Running prometheus on an LXC container on Proxmox VE. was getting the following error ->
WAL log samples: log series: write /var/lib/prometheus/wal/001530: disk quota exceeded

Because the disk was actually full. Later I increased the container size to 80GB from 32GB. and its reflected immediately.
rakesh.jain@labs-monitor:~$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
zfshome/images/subvol-111-disk-1 zfs 82G 32G 51G 39% /

But we still get the same error for the same old wal file.

daemon.log says ->
Aug 7 12:23:53 labs-monitor prometheus[21529]: level=warn ts=2018-08-07T12:23:53.076092475Z caller=scrape.go:713 component="scrape manager" scrape_pool=test-icmp target="http://labs-monitor.eng.fireeye.com:9115/probe?module=icmp&target=hpc123a.eng.fireeye.com" msg="append failed" err="WAL log samples: log series: write /var/lib/prometheus/wal/001530: disk quota exceeded"

here is the wal directory listing ->

rakesh.jain@labs-monitor:/var/lib/prometheus/wal$ ls -ltr
total 446203
-rw-r--r-- 1 prometheus prometheus 268428668 Jul 20 12:06 001527
-rw-r--r-- 1 prometheus prometheus 268434456 Jul 20 12:31 001528
-rw-r--r-- 1 prometheus prometheus 268428273 Jul 20 12:56 001529
-rw-r--r-- 1 prometheus prometheus 268435456 Jul 20 12:59 001530
-rw-r--r-- 1 prometheus prometheus 62241086 Aug 7 11:27 000001

rakesh.jain@labs-monitor:/var/lib/prometheus/wal$ du -sh *
6.8M 000001
137M 001527
137M 001528
137M 001529
19M 001530

[simonpasquier]

Thanks for your report. It looks as if this is actually a question about usage and not development because you're experiencing a file system problem with the underlying system.

To make your question, and all replies, easier to find, we suggest you move this over to our user mailing list, which you can also search. If you prefer more interactive help, join or our IRC channel, #prometheus on irc.freenode.net. Please be aware that our IRC channel has no logs, is not searchable, and that people might not answer quickly if they are busy or asleep. If in doubt, you should choose the mailing list.

Once your questions have been answered, please add a short line pointing to relevant replies in case anyone stumbles here via a search engine in the future.

[TroubleshooteRJ]

Sure. Thanks simonpasquier.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.