prometheus crashes

[zwopir]

prometheus crashes. No action except normal scaping during that time:
full panicparse stacktrace: https://gist.github.com/zwopiR/686590a0c53fd52ee8c9
truncated full stacktrace: https://gist.github.com/zwopiR/d00c6c63ce5ba4085973

prometheus, version 0.16.1 (branch: stable, revision: 968ee35)
build user: @9f8f0f8d724a
build date: 20160118-19:00:26
go version: 1.5.1
running inside docker prom/prometheus.
I happily provide any further information you may need

[brian-brazil]

Given that it failed creating a thread, the problem likely isn't on Prometheus's side. Did you run out of memory?

[zwopir]

node_memory_Active, node_memory_Cached compared to node_memory_MemTotal doesn't indicate so. Are they any metrics from node_exporter that may help tracking down this issue?

[brian-brazil]

What was node_memory_MemFree? If your Cached and MemFree is low you're out of memory.

[beorn7]

It could also be a rarely striking bug in Go1.5.1, which looks like you are out of memory. Try 0.16.2, which is compiled with Go1.5.3 or build it yourself with a recent Go compiler.

This is almost certainly not a bug in Prometheus itself.

[zwopir]

not sure about being out of memory. The the attached screenshot. The tooltip depicts roughly the last scrape, but anyway we'll try the new version

[brian-brazil]

That looks fine memory wise.

[zwopir]

so we might have hit the Go1.5.1 Bug?! we give the latest version a try. Thank you very much!

[beorn7]

@zwopir Any more crashes seen?

[zwopir]

we switched to 0.16.2 two days ago. No crashes so far, but this morning lots of
`time="2016-02-03T08:15:00Z" level=warning msg="Sample ingestion resumed." source="storage.go:555"
time="2016-02-03T08:15:00Z" level=warning msg="2097152 chunks waiting for persistence, sample ingestion suspended." source="storage.go:551"```

[fabxc]

This means that the load on your storage is somewhat high. You may want to consider switching to a larger server.

This sort of log-spam will be gone with 0.17.

[beorn7]

The sample ingestion suspended actually means you are in really bad shape. (It's not the 'storage is now in degraded mode' message, which is less bad than it sounds. Both issues are addressed in 0.17.)

You definitely need to either reduce your ingestion rate or increase the number of chunks allowed to wait for persistence even more (looks like you already did – if you have enough RAM, you can do so even more, see http://prometheus.io/docs/operating/storage/ for details).

Note that the number of chunks waiting for persistence is overestimated in 0.16.2. This is also fixed in 0.17.

[beorn7]

Closing since the original issue is resolved.

[zwopir]

thanks for the advice on storage parameters. Looking forward to 0.17

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.