graphite queue grows to ~10k or more after director deployment #8586
Comments
Tasks
|
|
Hi @marcelfischer, Thanks for the report. Can you also share some details here, how to get it reproduced, so, for example how often you deploy configs and if, you also have configured sync rules or only deploy single configs. I can't reproduce it unfortunately. Thanks! |
|
Currently we deploy once a day automatically via director. Some sync rules running before. Sometimes we have many changes, sometimes only a few custom variables for a bunch of hosts. But the issue is kind of independend of the amount of changes we activate. Every time we see the graphite queue and also the mysql queue having a huge spike. I think its also related to #5465 here you see a log output from our last deployment at 01:48 config master:
second master:
|
yhabteab
removed
the
needs feedback
|
@N-o-X Aren't you working on this at the moment? |
|
PING @N-o-X |
|
Nope, I didn't work on this or any related issues, sorry. |
|
@yhabteab How large was your reproducing setup? How much did it take to load the config? |
@marcelfischer Please share your config of those Icinga graphite features, also try v2.13.3+. |
Al2Klimov
added
the
needs feedback
Describe the bug
When we do a deployment in director, the queue for graphite in icinga2 goes to ~10k or even more. The queue itself goes back to 0 pretty fast but after that we typically see a lost of 10 to 15 minutes of performance data.
I wonder if this graphite queue does make sense anyway. Im not an graphite expert but as far as I understood, graphite or whisper automatically sets missing values to null. So if I expect a value every 2 minutes and icinga is currently reloading and queues some graphite data, this data wont fit into the whisper file anymore because its already too "late".
Two icinga masters with graphite in HA mode.
To Reproduce
Deploy Config with director, but I guess reloading icinga will lead to the same situation.
Expected behavior
I dont want to loose 10-15 Minutes of performance data.
Your Environment
Include as many relevant details about the environment you experienced the problem in
icinga2 --version): 2.11.8icinga2 feature list): api checker graphite ido-mysql mainlog notificationThe text was updated successfully, but these errors were encountered: