Prometheus frequently switch normal mode into rush mode

[Keleir]

My premehteus v1.7.1 occurs some error in log, prompts that the promtheus server enter into The Rush Mode frequencely:

time="2017-12-08T04:04:37+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=92489 memoryChunks=493796 source="storage.go:1867" urgencyScore=0.932 
time="2017-12-08T04:05:03+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=92439 memoryChunks=493196 source="storage.go:1857" urgencyScore=0.637 
time="2017-12-08T04:05:10+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=92413 memoryChunks=493327 source="storage.go:1867" urgencyScore=0.856 
time="2017-12-08T04:05:27+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=92473 memoryChunks=493225 source="storage.go:1857" urgencyScore=0.692 
time="2017-12-08T04:05:49+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=92633 memoryChunks=493890 source="storage.go:1867" urgencyScore=0.887 
time="2017-12-08T04:06:03+08:00" level=info msg="Storage has left rushed mode." chunksToPersist=92509 memoryChunks=493240 source="storage.go:1857" urgencyScore=0.695 
time="2017-12-08T04:06:15+08:00" level=warning msg="Storage has entered rushed mode." chunksToPersist=92421 memoryChunks=493468 source="storage.go:1867" urgencyScore=0.802 
time="2017-12-08T04:07:04+08:00" level=info msg="Storage does not need throttling anymore." chunksToPersist=94037 memoryChunks=495819 source="storage.go:1015" urgencyScore=0.952 
time="2017-12-08T04:08:08+08:00" level=error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped." chunksToPersist=94052 memoryChunks=494618 source="storage.go:1007" urgencyScore=1 

And there are also massive of hints of urgencyScore=1:

level=error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped."  
crapes and rule evaluations will be skipped." urgencyScore=1 

In my opinion the prometheus_local_storage_persistence_urgency_ score should be between 0 < urgencyScore < 1, when it bigger than 0.8 promeheus will enter rush mode.

Due to this, It seems some data lose that make some break points:

[simonpasquier]

Typically you would need to allocate more RAM to the Prometheus server to avoid this rush mode. Also it is better to ask such questions on the mailing list.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.