Exceeding max size of 4GiB #3190

tinytub · Sep 19, 2017

What did you do?
pull metric from my metric transfer
the transfer if translate over old monitor metrics to prometheus type
we have 250000metrics/minutes
after 12 hour running, the date dir is become about 700GB

what shold I do to solve this fail ?

What did you expect to see?
success compact

What did you see instead? Under which circumstances?
get a lot of compaction failed error

ts=2017-09-19T01:11:20.569106544Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T01:19:58.954523672Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T01:31:40.369418837Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T01:38:06.018397284Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T01:46:37.895183113Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:01:25.132604599Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:07:47.971239099Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:16:26.960117082Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:29:43.355890606Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:36:11.257146096Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:44:59.710578177Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"
ts=2017-09-19T02:56:59.32902664Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BTANR05SMQ4BW62TQBAGN1X7 /data1/prometheus/01BTB3MREQ6ZXGGM96W15KWRF9 /data1/prometheus/01BTBK2ZSGFQ0H79NYSGSZB5SZ]: write compaction: add series: write series data: exceeding max size of 4GiB"

Environment

System information:

Linux 2.6.32-696.3.2.el6.v6.2.x86_64 x86_64

Prometheus version:

prometheus, version 2.0.0-beta.4 (branch: HEAD, revision: 1b80f63)
build user: root@c5b3f10ffc51
build date: 20170914-11:22:50
go version: go1.8.3

gouthamve · Sep 19, 2017

That looks like some insane load, can you share more numbers as to how many hosts, services you are monitoring? If possible can you also provide the number of series, which can be found from the following metrics:
tsdb_head_series, tsdb_head_series_created_total, tsdb_head_series_removed_total.

AFAICS it can be 2 things: you have 50M+ series you are monitoring or you are putting a lot of info like verbose text into your labels. Are there any series which have panic traces or long strings?

tinytub · Sep 19, 2017

i have delete old data, when restart i got an error "Opening storage failed: head truncate failed: truncating at 1505788200000 not aligned"

I monitoring only 10 hosts but these hosts have total about 250000 series.
with the new clear start I change the --storage.tsdb.min-block-duration from 30m to 2h
labels is ok, and no panic trace or long string

I have start new prometheus process with clear date path, after one hour there is no new error log, I will monitor my promtheus host and report new errors i find.

thanks a lot

gouthamve · Sep 19, 2017

Hmm, 700G in 12hr for 250K series is very high.

Are the series constantly changing or are they mostly static?

tinytub · Sep 19, 2017

constantly changing.
the series from ten transfer host mostly like node monitor date from our old monitoring system. such as cpu, netdev, disk usage , etc. About 40000 hosts.

gouthamve · Sep 19, 2017

Oh damn, if I understand it right:

You are pushing several metrics from 40K hosts to 10 push gateway like systems.
This means you have waaay more than 250K series right?

This is a prometheus anti-pattern and you should look into scraping the hosts instead of pushing data. And I am not sure one Prometheus server can handle the load of 40K servers.

Can you get values of: tsdb_head_series, tsdb_head_series_created_total, tsdb_head_series_removed_total?

tinytub · Sep 19, 2017

about 3-4 hours
tsdb_head_series: 13281629
tsdb_head_series_created_total: 13614970
tsdb_head_series_removed_total: 333341

tinytub · Oct 11, 2017

upgrade to rc.0, still got the error "exceeding max size of 4GiB".
shold I change some parameter ？
--storage.tsdb.min-block-duration=2h
--storage.tsdb.max-block-duration=
this two param I use default value。

@gouthamve

fabxc · Oct 11, 2017

Did you close it again unintentionally. Could you share your logs?

tinytub · Oct 11, 2017

I closed the issue。
I find the error log come from my old prometheus beta.5 instance,not rc.0 .
This offset check also in rc.0 tsdb. my rc.0 just run two days, maybe i will got the error again after few days.

ts=2017-09-27T17:00:01.531646721Z caller=compact.go:359 msg="compact blocks" count=1 mint=1506520800000 maxt=1506528000000
ts=2017-09-27T17:02:57.353331372Z caller=head.go:261 msg="head GC completed" duration=12.047550015s
ts=2017-09-27T17:03:12.127797267Z caller=head.go:272 msg="WAL truncation completed" duration=14.774350046s
ts=2017-09-27T19:00:01.530925404Z caller=compact.go:359 msg="compact blocks" count=1 mint=1506528000000 maxt=1506535200000
ts=2017-09-27T19:03:00.257291055Z caller=head.go:261 msg="head GC completed" duration=12.903716109s
ts=2017-09-27T19:03:14.839752203Z caller=head.go:272 msg="WAL truncation completed" duration=14.582362192s
ts=2017-09-27T19:03:20.213820691Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506513600000 maxt=1506535200000
ts=2017-09-27T19:05:06.937210786Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T19:14:13.86104954Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T19:14:15.035277274Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T19:22:27.412462131Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T19:23:32.256732495Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T19:32:09.56322902Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T19:33:17.760325198Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T19:41:54.461240549Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T19:43:10.668707968Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T19:51:27.81299198Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T19:52:59.999685713Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T20:01:09.641499716Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T20:03:13.849396046Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T20:11:23.818212879Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T20:14:24.015105364Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T20:23:05.430583928Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T20:26:05.640274928Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T20:35:18.649508537Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T20:38:18.858056846Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T20:46:41.435941068Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T20:49:41.649806704Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T20:58:24.035102734Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T21:00:01.533730542Z caller=compact.go:359 msg="compact blocks" count=1 mint=1506535200000 maxt=1506542400000
ts=2017-09-27T21:02:55.95646049Z caller=head.go:261 msg="head GC completed" duration=12.908614762s
ts=2017-09-27T21:03:20.019700146Z caller=head.go:272 msg="WAL truncation completed" duration=24.063127265s
ts=2017-09-27T21:03:25.53030943Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T21:11:52.385191298Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T21:12:53.476090191Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T21:21:08.573080172Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T21:24:09.289739474Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T21:32:46.258963814Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T21:35:46.460969099Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T21:44:01.666213514Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T21:47:01.881739792Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T21:55:16.332135008Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T21:58:16.558468843Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T22:06:46.928373264Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T22:09:47.182227112Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T22:18:17.521433988Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T22:21:17.796085805Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T22:30:12.586923892Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T22:33:12.840904494Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T22:41:27.457944372Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T22:44:27.714798181Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T22:44:27.714798181Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T22:52:48.078377102Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T22:55:48.483097586Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T23:04:17.011524323Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T23:05:17.01250484Z caller=compact.go:359 msg="compact blocks" count=1 mint=1506542400000 maxt=1506549600000
ts=2017-09-27T23:09:50.433042122Z caller=head.go:261 msg="head GC completed" duration=29.218754864s
ts=2017-09-27T23:10:18.775347255Z caller=head.go:272 msg="WAL truncation completed" duration=28.3422036s
ts=2017-09-27T23:10:25.096920872Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T23:18:37.451714933Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T23:19:37.648232955Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T23:28:21.758765481Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T23:31:21.930193705Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T23:39:48.776567772Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T23:42:48.958551387Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-27T23:51:14.806203332Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-27T23:54:14.9924427Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T00:02:33.439965957Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T00:05:33.67026883Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T00:13:50.926496022Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T00:16:51.646082276Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T00:25:04.512167311Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T00:28:04.716389763Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T00:36:42.471380464Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T00:39:42.669980437Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T00:47:58.670024013Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T00:50:58.855378505Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T00:59:18.048238356Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T01:00:18.04922231Z caller=compact.go:359 msg="compact blocks" count=1 mint=1506549600000 maxt=1506556800000
ts=2017-09-28T01:03:37.95238233Z caller=head.go:261 msg="head GC completed" duration=14.595564824s
ts=2017-09-28T01:04:06.297778301Z caller=head.go:272 msg="WAL truncation completed" duration=28.345271402s
ts=2017-09-28T01:04:12.03829128Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506535200000 maxt=1506556800000
ts=2017-09-28T01:06:37.935157788Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T01:14:50.617923289Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T01:15:50.789936475Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T01:24:10.727912974Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T01:27:10.915476019Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T01:35:26.054848346Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T01:38:26.228378191Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T01:46:48.120986186Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T01:49:48.300055479Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T01:58:16.936722314Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T02:01:17.109324852Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T02:10:07.083750621Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T02:13:07.321136048Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T02:21:29.931316438Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T02:24:30.158550983Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T02:33:01.428044501Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write postings: write postings: exceeding max size of 4GiB"
ts=2017-09-28T02:36:01.688470493Z caller=compact.go:359 msg="compact blocks" count=3 mint=1506470400000 maxt=1506535200000
ts=2017-09-28T02:40:16.292476052Z caller=db.go:260 msg="compaction failed" err="compact [/data1/prometheus/01BV12NCQBF7X119N6RE2VRJ9T /data1/prometheus/01BV1PYH7NWAS07XPMH9TQHY2Y /data1/prometheus/01BV2BHMCNN9173A97PY5579B3]: write compaction: write chunks: no space left on device"
ts=2017-09-28T02:41:43.368123361Z caller=wal.go:638 msg="sync failed" err="flush buffer: write /data1/prometheus/wal/006945: file already closed"
ts=2017-09-28T02:41:53.368014847Z caller=wal.go:638 msg="sync failed" err="flush buffer: write /data1/prometheus/wal/006945: file already closed"
ts=2017-09-28T02:41:43.368123361Z caller=wal.go:638 msg="sync failed" err="flush buffer: write /data1/prometheus/wal/006945: file already closed"
ts=2017-09-28T02:41:53.368014847Z caller=wal.go:638 msg="sync failed" err="flush buffer: write /data1/prometheus/wal/006945: file already closed"

Got "exceeding max size of 4GiB" until my 1.4T disk full.

What cause my index larger than 4GiB? And what should I do to fix this?

fabxc · Oct 11, 2017

A high amount of series can actually cause this. We just did not anticipate that happening in practical setups to soon. If the issue reappears we might just extend our code to allow for larger file sizes.

We should also be more graceful about handling failed compactions by cleaning up the .tmp dirs. We might also want to mark failed ones to be skipped from retrying so we can go on compacting the those which do. @gouthamve

tinytub · Oct 12, 2017

hope for the new update.
should I reopen the issue ?

lueliu · Dec 14, 2017

I also encountered the same problem, but the data I monitored were not so much.
Prometheus version: v2.0.0
tsdb_head_series: 400355 （Historical peak：6450417）
tsdb_head_series_created_total: 954758
tsdb_head_series_removed_total: 554403

Last 30min:
rate(prometheus_tsdb_head_samples_appended_total[5m]) -- avg: 2.7k max/current: 2.9k
rate(prometheus_tsdb_head_series[5m]) -- avg: 198 max: 254 current: 179

Storage space will be taken up soon（2TB）
Usually, I can only restart the prometheus process and delete the directory of the suffix ".tmp".

The default parameter values are basically used at the moment. "storage.tsdb.min-block-duration" does the time of this parameter need to be set shorter?
Is it really because of my large amount of data that causes index to exceed 4GB?
my flags:
storage.tsdb.max-block-duration: 3d
storage.tsdb.min-block-duration: 2h

Prometheus Logs:

2017-12-14 13:01:06level=info ts=2017-12-14T05:01:06.004644044Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=97.734496ms
2017-12-14 13:01:34level=info ts=2017-12-14T05:01:34.102623329Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:06:18level=error ts=2017-12-14T05:06:18.251772964Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:09:41level=info ts=2017-12-14T05:09:41.118463658Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:14:26level=error ts=2017-12-14T05:14:26.888449455Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:17:49level=info ts=2017-12-14T05:17:49.712622132Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:22:35level=error ts=2017-12-14T05:22:35.051208756Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:25:58level=info ts=2017-12-14T05:25:58.796728783Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:30:46level=error ts=2017-12-14T05:30:46.728740034Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:34:07level=info ts=2017-12-14T05:34:07.828653574Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:39:00level=error ts=2017-12-14T05:39:00.750789998Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:42:21level=info ts=2017-12-14T05:42:21.477104657Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:47:10level=error ts=2017-12-14T05:47:10.443233966Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:50:31level=info ts=2017-12-14T05:50:31.75968018Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 13:55:19level=error ts=2017-12-14T05:55:19.194785515Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 13:58:40level=info ts=2017-12-14T05:58:40.296589974Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 14:03:28level=error ts=2017-12-14T06:03:28.656236071Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 14:06:49level=info ts=2017-12-14T06:06:49.554834288Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 14:11:45level=error ts=2017-12-14T06:11:45.187944822Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 14:15:06level=info ts=2017-12-14T06:15:06.370953724Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000
2017-12-14 14:20:00level=error ts=2017-12-14T06:20:00.064276415Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/prom/data/01C0TEWT5VAX17EFN0FVSBBJ2G /prom/data/01C0V3FNXMKA1DGXHS1FNSA6VC /prom/data/01C0VR1Z6J80WV3NSWSN1SK4VA]: close index writer: exceeding max size of 4GiB"
2017-12-14 14:23:21level=info ts=2017-12-14T06:23:21.388343101Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1512691200000 maxt=1512756000000

5.6G	01C19K6FSR86CQKNSYYW4K533G.tmp
5.6G	01C19KNE5STE38JGXQEP9R1R9V.tmp
5.6G	01C19M4AHNG24737Z4TTS0D5QP.tmp
5.6G	01C19MK894FESZXXFHGJABGM48.tmp
5.6G	01C19N25S02Y1G38DP42MAPNS1.tmp
5.6G	01C19NHDD4PKB9ZXM8G0KVC630.tmp
5.6G	01C19P0DA0V35PJPVSR9PWSHAC.tmp
12M	01C19PB7KG78MRY93QQTN0BBBJ
5.6G	01C19PCC0P0CMBCF56VZQ4WJCV.tmp
5.6G	01C19PV7KY9T5KA35P1H8REKYK.tmp
5.6G	01C19QA4RG8KP1TTA5GPDEE232.tmp
5.6G	01C19QS2CC6VWVJDA4MPMFV2TZ.tmp
5.6G	01C19R7ZYMMFASJ36NF8J9J6NV.tmp
5.6G	01C19RQ2159ZT5SZBFCC4F6JXJ.tmp
5.6G	01C19S60TF3XJTDK5VTZY9K1CZ.tmp
5.6G	01C19SMXX8F540VAY9FATFVAJ3.tmp
5.6G	01C19T3VPJX51QCQRZDWH1RRXE.tmp
5.6G	01C19TK0W2E63XTEKD1E7408HW.tmp
5.4G	01C19V249CPK7PS79A4YS7XR7K.tmp
4.0K	lock
515M	wal

lueliu · Dec 15, 2017

I compare my two Prometheus servers, although the Prom-A gets less metrics per second, but the dynamic time series is much more.

【samples_appended】
Prom-A：2000 metrics/s
Prom-B：5500 metrics/s
【max numSeries in meta.json】
Prom-A: 14037404（Data directory size 3GB+；When index exceeds 4GB, the data directory size 6GB+ is usually.）
Prom-B: 211770

Once the index exceeds the 4GB, the meta.json file is not generated.

The number of dynamically increased time series causes the size of the index file.

lueliu · Dec 15, 2017

Index files will grow to 2GB in 3 minutes

deejay1 · Dec 18, 2017

Same here on 2.0.0:

tsdb_head_series: 2162593
tsdb_head_series_created_total: 2162593
tsdb_head_series_removed_total: 0
rate(prometheus_tsdb_head_series[5m]): 0.00416
rate(prometheus_tsdb_head_samples_appended_total[5m]): 36043.216

du data

23G	./01C1F29D7DBB5PH42CRPQB8QVE
53G	./01BZV4C4W7ME7VTSHAR8XRF249
23G	./01C0XP36S6K1DTKQY7TNCW0EM0
4.3G	./wal
23G	./01C0C9Y4MYQ6D1S17DYSJS3DAB
267M	./01C1MEAVFW959YMRP1NSQCBMK6
40K	./01C1MAH1CN8957ZRWCFQS6BN9Q
458M	./01C1MAGAREN00V74RKP2DRD0ET
19G	./01C1MM67F028GVNYDZCCX49JTN.tmp
40K	./01C1MAG8WQQGDRFEMVX2JDZ4N8
40K	./01C1MAG6Z0PMAD13S5VWTWY784
144G	.

Runtime parameters

storage.tsdb.min-block-duration: 2h
storage.tsdb.max-block-duration: 52704m
storage.tsdb.retention: 366d

deejay1 · Dec 18, 2017

Dropping min/max block duration to 30m/1d respectively seems to have helped for now

lueliu · Dec 21, 2017

I've tried it too. It's very effective. THX. @deejay1

vnandha · Jan 9, 2018

I am also facing this issue on my federated cluster

Logs from the server:

level=info ts=2018-01-09T16:14:09.763451759Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T16:25:32.935897024Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T16:28:35.222435155Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T16:40:02.618381828Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T16:43:04.855224355Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T16:54:22.254932827Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T16:57:24.655014344Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T17:08:47.531711556Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T17:09:47.539511518Z caller=compact.go:361 component=tsdb msg="compact blocks" count=1 mint=1515506400000 maxt=1515513600000
level=info ts=2018-01-09T17:10:28.515455223Z caller=head.go:345 component=tsdb msg="head GC completed" duration=2.127481014s
level=info ts=2018-01-09T17:10:52.06063705Z caller=head.go:354 component=tsdb msg="WAL truncation completed" duration=23.545097818s
level=info ts=2018-01-09T17:10:55.13652097Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T17:22:04.281209871Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T17:23:06.583056994Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T17:34:19.694176112Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T17:37:21.891535494Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T17:48:35.917606861Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T17:51:38.206192495Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
level=error ts=2018-01-09T18:02:56.081874976Z caller=db.go:260 component=tsdb msg="compaction failed" err="compact [/var/prometheus/data/01C20EEVF22YRCFERX9GFRSVX5 /var/prometheus/data/01C2HTM6KT9BAQXH62VEW2EPX7 /var/prometheus/data/01C33
6TMWCMDVV8TM6MB23DR27]: write compaction: add series: write series data: exceeding max size of 4GiB"
level=info ts=2018-01-09T18:05:58.450458527Z caller=compact.go:361 component=tsdb msg="compact blocks" count=3 mint=1513404000000 maxt=1515153600000
-bash-4.2$ ^C
-bash-4.2$ kubectl exec -it -c prometheus-federation prometheus-k8s-federation-server-3440638662-635qg -n kube-monitoring sh

Tons of directories with .tmp 59G

/var/prometheus/data $ du -sh
4.3T    .
/var/prometheus/data $ du -sh *
19.3G   01C1F8W8E100E641XNE6HY6GVP
23.1G   01C20EEVF22YRCFERX9GFRSVX5
26.6G   01C2HTM6KT9BAQXH62VEW2EPX7
25.5G   01C336TMWCMDVV8TM6MB23DR27
3.0M    01C3C5ZYFVXRS7SJGAX4FYVC2Y
3.5G    01C3C600D2T4B2NE7JCEKC89HX
59.4G   01C3C62580VQDXXSW37V1TJK0Z.tmp
59.4G   01C3C6R7TCQ34NQDP3YSCXHKB0.tmp
59.4G   01C3C7EP3KER9TYS1VAC7X8B0H.tmp
59.4G   01C3C843CV499XHGQHYRYYFD87.tmp
59.4G   01C3C8VWMGWGSHJAKJ60MBMSX2.tmp
59.4G   01C3C9M1YKNPYG9RSN0BAZRAH9.tmp
59.4G   01C3CAD54NXVJE3S4DWSEJ3X76.tmp
59.4G   01C3CB8AF4G698E700AEGQFPD3.tmp
59.4G   01C3CC3P3YP0TQCJ6HSYA4T514.tmp
59.4G   01C3CCYYG9QJZW8XE8WWJP0401.tmp
59.4G   01C3CDTV7F6RE64T9RA4QMMQCP.tmp
59.4G   01C3CEJSC60RYP360SFGSMYQHV.tmp
59.4G   01C3CFE5NFXQ2QXQPSWD45BZKA.tmp

Inside 01C3CFE5NFXQ2QXQPSWD45BZKA.tmp

/var/prometheus/data/01C3CFE5NFXQ2QXQPSWD45BZKA.tmp $ du -sh *
55.4G   chunks
4.0G    index

lots of files with 512M inside chunks

-rw-r--r--    1 99       99        512.0M Jan  9 03:37 000105
-rw-r--r--    1 99       99        512.0M Jan  9 03:37 000106
-rw-r--r--    1 99       99        512.0M Jan  9 03:37 000107
-rw-r--r--    1 99       99        512.0M Jan  9 03:37 000108
-rw-r--r--    1 99       99        512.0M Jan  9 03:37 000109
-rw-r--r--    1 99       99        512.0M Jan  9 03:37 000110
-rw-r--r--    1 99       99        512.0M Jan  9 03:38 000111

Is this issue different from #3487 ?

vnandha · Jan 9, 2018

Is it safe to delete those .tmp directory? What is the impact of removing *.tmp directory

krasi-georgiev · Jan 9, 2018

I had a quick look recently about these and it seems that are created when running the compaction. unless Prometheus has crashed these shouldn't exist after the compaction is complete.

gouthamve · Jan 19, 2018

Closed in #3705

lock · Mar 23, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

tinytub changed the title ~~write compaction: add series: write series data: exceeding max size of 4GiB"~~ Exceeding max size of 4GiB Sep 19, 2017

tinytub closed this Sep 20, 2017

tinytub reopened this Oct 11, 2017

tinytub closed this Oct 11, 2017

gouthamve reopened this Dec 14, 2017

gouthamve referenced this issue Dec 28, 2017
Closed
Align series to 8/16byte padding to increase addressable space #238

simonpasquier referenced this issue Jan 12, 2018
Closed
'compaction failed' - prometheus suddenly ate up entire disk #3487

gouthamve closed this Jan 19, 2018

matejkramny referenced this issue Mar 21, 2018
Closed
Large chunks #3993

lock bot locked and limited conversation to collaborators Mar 23, 2019

prometheus/prometheus

Join GitHub today

Exceeding max size of 4GiB #3190

Comments

tinytub commented Sep 19, 2017 • edited

tinytub changed the title write compaction: add series: write series data: exceeding max size of 4GiB" Exceeding max size of 4GiB Sep 19, 2017

This comment has been minimized.

gouthamve commented Sep 19, 2017

This comment has been minimized.

tinytub commented Sep 19, 2017

This comment has been minimized.

gouthamve commented Sep 19, 2017

This comment has been minimized.

tinytub commented Sep 19, 2017

This comment has been minimized.

gouthamve commented Sep 19, 2017

This comment has been minimized.

tinytub commented Sep 19, 2017 • edited

tinytub closed this Sep 20, 2017

This comment has been minimized.

tinytub commented Oct 11, 2017 • edited

tinytub reopened this Oct 11, 2017

tinytub closed this Oct 11, 2017

This comment has been minimized.

fabxc commented Oct 11, 2017

This comment has been minimized.

tinytub commented Oct 11, 2017

This comment has been minimized.

fabxc commented Oct 11, 2017

This comment has been minimized.

tinytub commented Oct 12, 2017

This comment has been minimized.

lueliu commented Dec 14, 2017

gouthamve reopened this Dec 14, 2017

This comment has been minimized.

lueliu commented Dec 15, 2017

This comment has been minimized.

lueliu commented Dec 15, 2017

This comment has been minimized.

deejay1 commented Dec 18, 2017 • edited

This comment has been minimized.

deejay1 commented Dec 18, 2017

This comment has been minimized.

lueliu commented Dec 21, 2017

gouthamve referenced this issue Dec 28, 2017

Align series to 8/16byte padding to increase addressable space #238

This comment has been minimized.

vnandha commented Jan 9, 2018

This comment has been minimized.

vnandha commented Jan 9, 2018 • edited

This comment has been minimized.

krasi-georgiev commented Jan 9, 2018 • edited

simonpasquier referenced this issue Jan 12, 2018

'compaction failed' - prometheus suddenly ate up entire disk #3487

This comment has been minimized.

gouthamve commented Jan 19, 2018

gouthamve closed this Jan 19, 2018

matejkramny referenced this issue Mar 21, 2018

Large chunks #3993

This comment has been minimized.

lock bot commented Mar 23, 2019

lock bot locked and limited conversation to collaborators Mar 23, 2019

tinytub commented Sep 19, 2017 •

edited

tinytub commented Sep 19, 2017 •

edited

tinytub commented Oct 11, 2017 •

edited

deejay1 commented Dec 18, 2017 •

edited

vnandha commented Jan 9, 2018 •

edited

krasi-georgiev commented Jan 9, 2018 •

edited