Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prometheus server crash with OOM #2300

Closed
jinhang opened this Issue Dec 22, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@jinhang
Copy link

jinhang commented Dec 22, 2016

time="2016-12-21T16:01:47+08:00" level=info msg="Done checkpointing in-memory metrics and chunks in 8.6828682s." source="persistence.go:573"
runtime: out of memory: cannot allocate 720896-byte block (1342177280 in use)
fatal error: out of memory

time="2016-12-21T16:16:15+08:00" level=error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped." chunksToPersist=433329 maxChunksToPersist=536870912 maxToleratedMemChunks=-1181116006 memoryChunks=467574 source="storage.go:895"
panic: runtime error: makeslice: len out of range

goroutine 163 [running]:
panic(0xbf1ea0, 0x46935148)
/usr/local/go/src/runtime/panic.go:500 +0x331
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).maybeEvict(0x1379a240)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1057 +0x62
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).handleEvictList(0x1379a240)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1022 +0x168
created by github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:367 +0x348

@jinhang

This comment has been minimized.

Copy link
Author

jinhang commented Dec 22, 2016

windows : 16RAM mem
start prometheus with default parameters

@juliusv

This comment has been minimized.

Copy link
Member

juliusv commented Dec 23, 2016

Is this the same as #2299? Why did you close that one without a comment and opened a new one? Please be considerate in how you report and deal with issues.

Again, as stated in #2299, your problem is that you are setting the -storage.local.memory-chunks flag value way too high (then Prometheus thinks it should keep way too many chunks in memory, but your machine does not have infinite memory). Try setting it to a couple of hundred thousands or maybe a few millions at most, with your 16GB of memory.

@juliusv juliusv closed this Dec 23, 2016

@juliusv

This comment has been minimized.

Copy link
Member

juliusv commented Dec 23, 2016

Ehm, I meant #2293 which was closed by you. The other one was closed by someone else. Still, not helpful to have three issues about the same thing.

@lock

This comment has been minimized.

Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.