disk-buffer: add metrics for abandoned disk-buffer files #4402

alltilla · 2023-03-28T13:50:16Z

Availability is the same as the disk_queue_dir_available_bytes metric.

Example metrics:

syslogng_disk_queue_capacity_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 104853504
syslogng_disk_queue_disk_allocated_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 273408
syslogng_disk_queue_disk_usage_bytes{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 269312
syslogng_disk_queue_events{abandoned="true",path="/var/syslog-ng/syslog-ng-00000.rqf",reliable="true"} 860

Depends on #4399 and #4356.

Signed-off-by: Attila Szakacs attila.szakacs@axoflow.com

Fixes #4400

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

kira-syslogng · 2023-03-28T14:56:50Z

Build FAILURE

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

kira-syslogng · 2023-03-30T11:43:27Z

Build FAILURE

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla · 2023-03-30T20:15:36Z

@kira-syslogng do stresstest

kira-syslogng · 2023-03-30T20:39:48Z

Kira-stress-test: Build SUCCESS

MrAnno

Nice feature.

For the record: diskq-global-metrics considers buffers abandoned when they are not used in the currently running syslog-ng instance. This is a bit different than what we do in our persist file: we abandon buffers only when 2 consecutive syslog-ng restart does not acquire them.

I think this is not a real problem, I just wanted to mention it.

modules/diskq/logqueue-disk-non-reliable.c

modules/diskq/diskq-global-metrics.c

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla · 2023-04-04T19:12:13Z

Rebased to master and fixed all the comments I think. I have not tested it manually yet, but I will do that tomorrow.

MrAnno · 2023-04-04T20:27:19Z

@kira-syslogng do stresstest

kira-syslogng · 2023-04-04T20:51:50Z

Kira-stress-test: Build SUCCESS

alltilla · 2023-04-05T08:42:12Z

Tested manually, the behavior looks good to me.

lib/stats/stats-registry.c

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla · 2023-04-06T13:09:58Z

@kira-syslogng do stresstest

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

For force removing clusters. Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

Asserting seems a but too strict. As the function can return error, it is enought to use that. Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla · 2023-04-06T13:31:24Z

I have loosened an assert in QDisk, as I am a bit paranoid that if the file gets emptied by something external in a bad split second, we could fail that assert.

kira-syslogng · 2023-04-06T13:34:23Z

Kira-stress-test: Build SUCCESS

alltilla · 2023-04-06T13:40:30Z

@kira-syslogng do stresstest

kira-syslogng · 2023-04-06T14:05:01Z

Kira-stress-test: Build SUCCESS

MrAnno · 2023-04-11T10:17:53Z

I'm investigating a SIGABRT in disk-buffer when config initialization is not successful (I'm not sure this is related to this PR).

Update: unrelated.

MrAnno

LGTM.

modules/diskq/qdisk.c

lib/stats/stats-registry.c

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

czanik · 2023-04-11T13:00:17Z

@alltilla Build on SLES 12 verified. Thanks!

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 28, 2023

news: add entry for syslog-ng#4402

55de645

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 1a0a967 to 55de645 Compare March 28, 2023 14:00

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 28, 2023

news: add entry for syslog-ng#4402

fb0ce04

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 55de645 to fb0ce04 Compare March 28, 2023 14:16

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 28, 2023

news: add entry for syslog-ng#4402

55109b7

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from fb0ce04 to 55109b7 Compare March 28, 2023 14:34

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 28, 2023

news: add entry for syslog-ng#4402

a78f92f

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 55109b7 to a78f92f Compare March 28, 2023 15:33

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 30, 2023

news: add entry for syslog-ng#4402

04c222a

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from a78f92f to 04c222a Compare March 30, 2023 11:22

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 30, 2023

news: add entry for syslog-ng#4402

b53df5e

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 04c222a to b53df5e Compare March 30, 2023 11:51

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 30, 2023

news: add entry for syslog-ng#4402

26b05a5

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from b53df5e to 26b05a5 Compare March 30, 2023 16:51

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Mar 30, 2023

news: add entry for syslog-ng#4402

58b374e

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 26b05a5 to 58b374e Compare March 30, 2023 20:14

MrAnno self-requested a review April 2, 2023 22:24

MrAnno reviewed Apr 4, 2023

View reviewed changes

modules/diskq/diskq-global-metrics.c Outdated Show resolved Hide resolved

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Apr 4, 2023

news: add entry for syslog-ng#4402

bc07f9f

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 58b374e to bc07f9f Compare April 4, 2023 19:08

alltilla commented Apr 5, 2023

View reviewed changes

lib/stats/stats-registry.c Show resolved Hide resolved

alltilla added 4 commits April 6, 2023 15:05

qdisk: add disk-buffer filename validator

5054e0b

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

qdisk: add disk-buffer filename reliable checker

e2eac4d

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: sweep through files in dirs

070b246

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: add initializator for abandoned metric keys

c3e78eb

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Apr 6, 2023

news: add entry for syslog-ng#4402

8c5a58d

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 678366c to 8c5a58d Compare April 6, 2023 13:05

alltilla added 7 commits April 6, 2023 15:22

diskq-global-metrics: set metrics for abandoned files

5bc57a6

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

stats-registry: add stats_remove_cluster()

a68ff12

For force removing clusters. Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: unset metrics for reacquired files

0ca3c45

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: factor out _init_dir_sc_keys()

f601952

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: unset all metrics in deinit()

098c1ec

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: add lock related comments to functions

802ae49

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

diskq-global-metrics: change error level to debug

64c8a72

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Apr 6, 2023

news: add entry for syslog-ng#4402

f5be6f8

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from 8c5a58d to f5be6f8 Compare April 6, 2023 13:22

alltilla added 2 commits April 6, 2023 15:29

qdisk: do not assert non-read-only in create_file()

3105c53

Asserting seems a but too strict. As the function can return error, it is enought to use that. Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

news: add entry for syslog-ng#4402

1235215

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

alltilla force-pushed the disk-buffer-abandoned-metrics branch from f5be6f8 to 1235215 Compare April 6, 2023 13:29

MrAnno self-requested a review April 6, 2023 15:29

MrAnno reviewed Apr 11, 2023

View reviewed changes

modules/diskq/qdisk.c Show resolved Hide resolved

lib/stats/stats-registry.c Show resolved Hide resolved

MrAnno approved these changes Apr 11, 2023

View reviewed changes

MrAnno merged commit cc1a3a2 into syslog-ng:master Apr 11, 2023

alltilla added a commit to alltilla/syslog-ng that referenced this pull request Apr 11, 2023

news: add entry for syslog-ng#4402

7be28c2

Signed-off-by: Attila Szakacs <attila.szakacs@axoflow.com>

disk-buffer: add metrics for abandoned disk-buffer files #4402

disk-buffer: add metrics for abandoned disk-buffer files #4402

Uh oh!

Conversation

alltilla commented Mar 28, 2023 • edited by MrAnno Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kira-syslogng commented Mar 28, 2023

Uh oh!

kira-syslogng commented Mar 30, 2023

Uh oh!

alltilla commented Mar 30, 2023

Uh oh!

kira-syslogng commented Mar 30, 2023

Uh oh!

MrAnno left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alltilla commented Apr 4, 2023

Uh oh!

MrAnno commented Apr 4, 2023

Uh oh!

kira-syslogng commented Apr 4, 2023

Uh oh!

alltilla commented Apr 5, 2023

Uh oh!

Uh oh!

alltilla commented Apr 6, 2023

Uh oh!

alltilla commented Apr 6, 2023

Uh oh!

kira-syslogng commented Apr 6, 2023

Uh oh!

alltilla commented Apr 6, 2023

Uh oh!

kira-syslogng commented Apr 6, 2023

Uh oh!

MrAnno commented Apr 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MrAnno left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

czanik commented Apr 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alltilla commented Mar 28, 2023 •

edited by MrAnno

Loading

MrAnno commented Apr 11, 2023 •

edited

Loading