Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement WorkQueue metric stats and periodic logging #5280

Merged
merged 1 commit into from May 23, 2017

Conversation

dnsmichi
Copy link
Contributor

@dnsmichi dnsmichi commented May 22, 2017

This will only log an entry if

  • there are pending tasks inside the work queue
  • 5 min interval after startup

There's also the current rate calculation and "queue empty" estimation as you know from the IDO feature already.

This PR also solves the problem of spamming the log with empty queues every 10 seconds.

Any feature which uses a work queue in the future will get those metrics and logs "for free".

Bonus: Rate calculation is available to any work queue owner, so it can be used for API and built-in performance data metrics too in #5266 and #5133.

# icinga2 daemon | grep WorkQueue
[2017-05-23 15:56:26 +0200] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2017-05-23 15:56:26 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 5, rate:  0/s (0/min 0/5min 0/15min); empty in 17309 days, 13 hours, 56 minutes and 26 seconds
[2017-05-23 15:56:26 +0200] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 158.95/s (9537/min 9537/5min 9537/15min);
[2017-05-23 15:56:26 +0200] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 0, rate: 3.81667/s (229/min 229/5min 229/15min);
[2017-05-23 15:56:36 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 18, rate:  0/s (0/min 0/5min 0/15min); empty in 13 seconds
[2017-05-23 15:56:46 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 31, rate:  0/s (0/min 0/5min 0/15min); empty in 23 seconds
[2017-05-23 15:56:56 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 940, rate:  0/s (0/min 0/5min 0/15min); empty in 10 seconds
[2017-05-23 15:57:06 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 952, rate:  0/s (0/min 0/5min 0/15min); empty in 13 minutes and 13 seconds
[2017-05-23 15:57:16 +0200] information/WorkQueue: #5 (InfluxdbWriter, influxdb) items: 965, rate:  0/s (0/min 0/5min 0/15min); empty in 12 minutes and 22 seconds

@dnsmichi dnsmichi self-assigned this May 22, 2017
@dnsmichi dnsmichi added area/api REST API area/distributed Distributed monitoring (master, satellites, clients) enhancement New feature or request area/log Logging related labels May 22, 2017
@dnsmichi dnsmichi force-pushed the feature/workqueue-stats branch 2 times, most recently from c0bcfcf to fc726a0 Compare May 23, 2017 13:48
@dnsmichi dnsmichi changed the title WIP: Implement WorkQueue stats and logging Implement WorkQueue stats and periodic logging May 23, 2017
@dnsmichi dnsmichi added this to the 2.7.0 milestone May 23, 2017
@dnsmichi dnsmichi changed the title Implement WorkQueue stats and periodic logging Implement WorkQueue metric stats and periodic logging May 23, 2017
@dnsmichi dnsmichi merged commit cbec910 into master May 23, 2017
@dnsmichi dnsmichi deleted the feature/workqueue-stats branch May 23, 2017 14:30
dnsmichi pushed a commit that referenced this pull request May 24, 2017
This also adds reconnect handling and exceptions.

refs #5132
refs #5133
refs #5280
dnsmichi pushed a commit that referenced this pull request May 26, 2017
This also adds reconnect handling and exceptions.

refs #5132
refs #5133
refs #5280
dnsmichi pushed a commit that referenced this pull request May 26, 2017
This also adds reconnect handling and exceptions.

refs #5132
refs #5133
refs #5280
dnsmichi pushed a commit that referenced this pull request May 26, 2017
This also adds reconnect handling and exceptions.

refs #5132
refs #5133
refs #5280
dnsmichi pushed a commit that referenced this pull request May 26, 2017
Use workqueues in Graphite and InfluxDB features

refs #5133
refs #5280
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/api REST API area/distributed Distributed monitoring (master, satellites, clients) area/log Logging related enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant