cfg: add named logpaths with counters by alltilla · Pull Request #4344 · syslog-ng/syslog-ng

alltilla · 2023-02-24T14:02:57Z

E.g.:

log top-level {
    source(s_local);

    log inner-1 {
        filter(f_inner_1);
        destination(d_local_1);
    };

    log inner-2 {
        filter(f_inner_2);
        destination(d_local_2);
    };
};

Each named log path counts its ingress and egress messages:

syslogng_log_path_ingress{id="top-level"} 114
syslogng_log_path_ingress{id="inner-1"} 114
syslogng_log_path_ingress{id="inner-2"} 114
syslogng_log_path_egress{id="top-level"} 103
syslogng_log_path_egress{id="inner-1"} 62
syslogng_log_path_egress{id="inner-2"} 41

Note that the egress statistics only count the messages which have been have not been filtered out from the related
log path, it does care about whether there are any destinations in it or that any destination delivers or drops the message.

Depends on #4325 and #4339.

Signed-off-by: Attila Szakacs szakacs.attila96@gmail.com

github-actions · 2023-02-24T14:03:13Z

No news file has been detected. Please write one, if applicable.

It makes a neater output IMO. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

alltilla · 2023-03-01T09:27:50Z

Updated the E2E tests according to what we agreed on.

kira-syslogng · 2023-03-02T16:04:46Z

Build FAILURE

MrAnno

Nice!

I would like to look into the commit [cfg-tree: inject metrics-pipes to named log paths](https://github.com/syslog-ng/syslog-ng/pull/4344/commits/e572f54b4eaf06b82d2fe8a9a94c5cbac6bbb88d) more closely, I'll get back with an approve or another review note in a few hours.

MrAnno · 2023-03-03T11:19:12Z

@kira-syslogng do stresstest

kira-syslogng · 2023-03-03T11:26:29Z

Kira-stress-test: Build FAILURE

MrAnno · 2023-03-03T11:42:05Z

@HofiOne Kira is broken due to an internal Java change. Can you check it, please?

kira-syslogng · 2023-03-03T12:47:19Z

Build FAILURE

HofiOne · 2023-03-03T14:31:46Z

@HofiOne Kira is broken due to an internal Java change. Can you check it, please?

done, sorry for the inconvenience

MrAnno · 2023-03-03T14:40:23Z

@kira-syslogng retest this please

MrAnno · 2023-03-03T14:40:35Z

@kira-syslogng do stresstest

@HofiOne Thanks

MrAnno · 2023-03-03T15:01:55Z

LGTM otherwise.

kira-syslogng · 2023-03-03T15:05:00Z

Kira-stress-test: Build SUCCESS

bazsi · 2023-03-03T20:30:02Z

+  self->log_path_name = g_strdup(log_path_name);
+
+  log_pipe_add_info(&self->super, "metrics-pipe");
+  log_pipe_add_info(&self->super, self->log_path_name);


We should start specifying some kind of "type" qualifier to info, so that we could use the same mechanism for the debugger, but in a way that allows programmatic use.

Like use a key-value pair here.

This is more like a where should we go next kind of comment, not to be addressed here.

I agree, we could add more detailed info for the LogPipes. Let's gather the use cases and make the LogPipe info functionality smarter.

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

MrAnno · 2023-03-06T17:00:04Z

This is my last comment, I promise. LGTM otherwise.

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

The ingress counter is incremented once, when its queue() method is called, the egress counter is increased once, when the message gets matched by at least one of the following pipes. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

alltilla force-pushed the named-log-paths branch 2 times, most recently from 8a90734 to 3399d17 Compare February 27, 2023 08:04

MrAnno self-requested a review February 27, 2023 08:59

MrAnno added this to the syslog-ng 4.1 milestone Feb 27, 2023

config-graph-json-to-dot: use boxes for nodes

a9ae8db

It makes a neater output IMO. Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

alltilla force-pushed the named-log-paths branch 2 times, most recently from c017eeb to 4044b4e Compare March 1, 2023 09:26

alltilla force-pushed the named-log-paths branch 4 times, most recently from 949234f to 2d374ea Compare March 2, 2023 15:42

MrAnno reviewed Mar 3, 2023

View reviewed changes

Comment thread lib/cfg-grammar.y Outdated

Comment thread lib/cfg-tree.c

Comment thread lib/metrics-pipe.c Outdated

alltilla force-pushed the named-log-paths branch from 2d374ea to 9719cf0 Compare March 3, 2023 12:24

MrAnno reviewed Mar 3, 2023

View reviewed changes

Comment thread lib/cfg-grammar.y

bazsi reviewed Mar 3, 2023

View reviewed changes

Comment thread lib/cfg-grammar.y Outdated

bazsi reviewed Mar 3, 2023

View reviewed changes

cfg: make it possible to name a logpath

a3a1842

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

alltilla force-pushed the named-log-paths branch from 32851fe to 55d30dc Compare March 6, 2023 09:04

MrAnno reviewed Mar 6, 2023

View reviewed changes

Comment thread lib/cfg-tree.c Outdated

alltilla added 13 commits March 6, 2023 19:03

cfg: disallow setting logpaths with the same name

eb62334

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

cfg-tree: inject metrics-pipes to named log paths

cd3a5da

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add option to create named logpath

6113397

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: fix indentation of inner logpaths

9797a03

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add support for syslog-ng-ctl stats prometheus

9735c33

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add prometheus stats support for LogPath

d11efd4

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add stats assert helper to logpath functional tests

9b33d33

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add E2E tests for named logpaths without flags

56f17f2

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add E2E tests for named logpaths with catchall flag

d7cf0b8

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add E2E tests for named logpaths with fallback flag

e589306

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

light: add E2E tests for named logpaths with final flag

668b218

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

news: add entry for syslog-ng#4342

e2c0fb1

Signed-off-by: Attila Szakacs <szakacs.attila96@gmail.com>

alltilla force-pushed the named-log-paths branch from 55d30dc to e2c0fb1 Compare March 6, 2023 18:05

MrAnno approved these changes Mar 6, 2023

View reviewed changes

MrAnno merged commit aafe466 into syslog-ng:master Mar 6, 2023

Conversation

alltilla commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Feb 24, 2023

Uh oh!

alltilla commented Mar 1, 2023

Uh oh!

kira-syslogng commented Mar 2, 2023

Uh oh!

MrAnno left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MrAnno commented Mar 3, 2023

Uh oh!

kira-syslogng commented Mar 3, 2023

Uh oh!

MrAnno commented Mar 3, 2023

Uh oh!

kira-syslogng commented Mar 3, 2023

Uh oh!

Uh oh!

HofiOne commented Mar 3, 2023

Uh oh!

MrAnno commented Mar 3, 2023

Uh oh!

MrAnno commented Mar 3, 2023

Uh oh!

MrAnno commented Mar 3, 2023

Uh oh!

kira-syslogng commented Mar 3, 2023

Uh oh!

Uh oh!

bazsi Mar 3, 2023

Choose a reason for hiding this comment

Uh oh!

bazsi Mar 4, 2023

Choose a reason for hiding this comment

Uh oh!

alltilla Mar 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MrAnno commented Mar 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

alltilla commented Feb 24, 2023 •

edited

Loading