Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

logqueue: add queue level metrics #4392

Merged
merged 3 commits into from Mar 30, 2023
Merged

logqueue: add queue level metrics #4392

merged 3 commits into from Mar 30, 2023

Conversation

alltilla
Copy link
Collaborator

@alltilla alltilla commented Mar 23, 2023 •
edited

This PR introduces queue level metrics:

  • The corresponding driver is identified with the "id" and "driver_instance" labels.
  • Available counters are "memory_usage_bytes" and "events".
  • Memory queue metrics are available with "syslogng_memory_queue_" prefix,
    disk-buffer metrics are available with "syslogng_disk_queue_" prefix.
  • disk-buffer metrics have an additional "path" label, pointing to the location of the disk-buffer file
    and a "reliable" label, which can be either "true" or "false".
  • Threaded destinations, like http, python, etc have an additional "worker" label.

Example metrics

syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 80
syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 7
syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 7
syslogng_disk_queue_events{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 7
syslogng_disk_queue_events{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 101
syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true",worker="0"} 3136
syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00001.rqf",reliable="true",worker="1"} 2776
syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00002.rqf",reliable="true",worker="2"} 2760
syslogng_disk_queue_memory_usage_bytes{driver_instance="http,http://localhost:1239",id="d_http_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00003.rqf",reliable="true",worker="3"} 2776
syslogng_disk_queue_memory_usage_bytes{driver_instance="tcp,localhost:1235",id="d_network_disk_buffer#0",path="/var/syslog-ng/syslog-ng-00000.qf",reliable="false"} 39888
syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 15
syslogng_memory_queue_events{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 14
syslogng_memory_queue_events{driver_instance="tcp,localhost:1234",id="d_network#0"} 29
syslogng_memory_queue_memory_usage_bytes{driver_instance="http,http://localhost:1236",id="d_http#0",worker="0"} 5896
syslogng_memory_queue_memory_usage_bytes{driver_instance="http,http://localhost:1236",id="d_http#0",worker="1"} 5552
syslogng_memory_queue_memory_usage_bytes{driver_instance="tcp,localhost:1234",id="d_network#0"} 11448

The values of these counters are unit tested. Before this PR, they were only atomic gssizes, now they are real StatsCounterItems, and the existing unit tests tests their values. I tried to add E2E tests, but it became really complex to do with Light, and I did not see the big added benefit.

Depends on #4389.

Signed-off-by: Attila Szakacs attila.szakacs@axoflow.com

@alltilla
Copy link
Collaborator Author

@kira-syslogng do stresstest

@kira-syslogng
Copy link
Contributor

Kira-stress-test: Build SUCCESS

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 24, 2023
@alltilla
news: add entry for syslog-ng#4392
086b11b 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla alltilla force-pushed the logqueue-add-queue-level-metrics branch from 38c5cb7 to 086b11b Compare March 24, 2023 11:38
@alltilla alltilla mentioned this pull request Mar 24, 2023
Merged
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 28, 2023
@alltilla
news: add entry for syslog-ng#4392
b626986 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla alltilla force-pushed the logqueue-add-queue-level-metrics branch from 086b11b to b626986 Compare March 28, 2023 15:52
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 29, 2023
@alltilla
news: add entry for syslog-ng#4392
cc08cd2 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla alltilla force-pushed the logqueue-add-queue-level-metrics branch from b626986 to cc08cd2 Compare March 29, 2023 13:19
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 29, 2023
@alltilla
news: add entry for syslog-ng#4392
b4cd638 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla alltilla force-pushed the logqueue-add-queue-level-metrics branch from cc08cd2 to b4cd638 Compare March 29, 2023 13:28
@MrAnno MrAnno self-requested a review March 30, 2023 13:29
@alltilla
stats-cluster-key-builder: add set_name_prefix()
33e7c7c 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 30, 2023
@alltilla
news: add entry for syslog-ng#4392
89f556a 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla alltilla force-pushed the logqueue-add-queue-level-metrics branch from b4cd638 to 89f556a Compare March 30, 2023 14:19
@alltilla
logqueue: add queue level metrics
17853d8 
With this commit, the atomic counters turned into StatsCounterItems.

Similarly to the driver level metrics, the queue level metrics' sc keys
are initialized with the Builder class, but the data is built up on
different layers:
  * LogWriter and LogThreadedDestDriver initializes the labels for the
    driver reference ("id" and "driver_instance").
  * LogThreadedDestDriver adds a "worker" label.
  * LogQueueFifo and LogQueueDisk sets the name prefix to
    "memory_queue_" and "disk_queue_".
  * LogQueueDisk adds a "path" label, pointing to the disk-buffer
    file's location and a "reliable" label.
  * LogQueue registers the "memory_usage_bytes" and "events" counters.

Queue level metrics are enabled from stats(level(1)).

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla
news: add entry for syslog-ng#4392
ef44b63 
Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>
@alltilla alltilla force-pushed the logqueue-add-queue-level-metrics branch from 89f556a to ef44b63 Compare March 30, 2023 14:23
@alltilla
Copy link
Collaborator Author

Rebased to master, fixed the comments and removed 2 accidentally left in printf()s.

@alltilla
Copy link
Collaborator Author

@kira-syslogng do stresstest

@kira-syslogng
Copy link
Contributor

Kira-stress-test: Build SUCCESS

@MrAnno MrAnno merged commit 5c7784c into syslog-ng:master Mar 30, 2023
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MrAnno MrAnno MrAnno approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@alltilla @kira-syslogng @MrAnno