syslog-ng 4

[bazsi]

This discussion is to collect changes we plan for syslog-ng 4

[bazsi]

Java upgrade from java8 to 11?

It proved to be difficult to support old java (less than) and newer ones. If we could change the minimum requirement to a newer stack, we would be able to drop compatibility with the older versions. See #3432

Change:
We would drop compatibility with the old versions of Java.

[bazsi]

1. We won't wait for a solution with 4.0 (that means that we would release 4.0 even if no solution surfaces)
2. if no solution becomes available until 4.0, we would include a deprecation warning in the NEWS file (unless a volunteer comes up)

[MrAnno]

Hopefully, #4054 will be a solution.

[bazsi]

Type aware comparisons

We have two sets of comparison operators, numeric ones (<, > and =) and string based ones (lt, gt, and eq). This is confusing at first, and also the seemingly trivial expression "$foo" == "" is always true if $foo is not a number.

$foo is converted to a number, which becomes 0, the empty string is converted to a number, also 0, the result of the comparison is TRUE, while $foo was not an empty string.

Fix this by interpreting the operators just like JavaScript does: try to do the right thing. strings compare as strings, numbers compare as numbers.

Change:
the configuration would interpret comparisons differently, even though the new behaviour makes more sense.

[MrAnno]

Under review.

[bazsi]

This is merged!

[bazsi]

Retain type information for name-value pairs

Some of our inputs are typed (e.g. JSON), retain this type information when the same name-value pair is reproduced at the output side.

Change:
This can cause changes in our output, as some JSON attributes would become typed while previously they were always strings.

[bazsi]

This has been merged for 3.37 already.

[bazsi]

Make some of our macros typed

There are macros that are numeric in nature $MIN or $YEAR or $FACILITY_NUM. Make these typed, just like name-value pairs.

Change:
whenever we use these macros in type aware outputs (e.g. $(format-json)), the previously string values would become numeric.

[bazsi]

This has been merged for 3.37 already.

[bazsi]

Add typing support to these components:

• db-parser()/grouping-by(): values in synthetic message should set name-values with types
• add-contextual-data(): set typed values when using templates
• python(): support for typing in the LogMessage wrapper, value-pairs?
• riemann(): riemann_dd_field_integer_maybe_add to use type-cast/generic-number
• SQL destination
• groupset()
• filter expressions

[bazsi]

For add-contextual-data, a new PR has just been opened (#4051)

[bazsi]

The filter expressions item is covered by #4022

[bazsi]

Python typing support is implemented in #4057

[bazsi]

Refactor: Numeric functions should use generic-number.{c,h} to convert input to a numeric representation.

[MrAnno]

Under review

[MrAnno]

Merged.

[bazsi]

Refactor: "str-repr" library

syslog-ng typing works so that name-value pairs contain a string representation of a specific type. Also, we store the "type" information along with this string.

Whenever syslog-ng needs something as a specific type, it consults the type information first (e.g. consider that the name-value pair actually stores a number or a list). Then once the type is established, it "parses" the string based representation in a specific way (lists are represented as properly escaped strings).

Today, the code to parse or reproduce the types in question is scattered around the code base:

• strings are decoded/encoded by functions in lib/str-repr
• numbers are using the generic-number.{c,h} module (not yet on master)
• lists are decoded using list-scanner, encoded using open-coded code snippets calling str-repr
• and there are type hinting helpers

It would make sense to concentrate all of the "string-representation" code in lib/str-repr, creating an consistent API along the various types we can have in syslog-ng. Probably the type-casting helpers would find a better home here.

[MrAnno]

Nice-to-have

[bazsi]

Improve warning reporting

Warnings are reported either during parsing, or later in init() functions. While error messages are quite readable and produce a very nice context of the error, warning messages contain a location information (file name, row/column) at best.

With the anticipated changes in 4.0, I expect to have a larger number of upgrade warnings, so it would be great if those were a bit more easier to grasp and act upon.

[MrAnno]

Nice-to-have.

[MrAnno]

It sounds scary, but maybe we could enable flow-control as a default to be "reliable by default", and add options for disabling it.
An exception could be the system() source (not the entire log path, just the source), so we won't completely block /dev/log.

[MrAnno]

We decided not to do that.

[MrAnno]

Enable RFC5424 parsing for the network() sources

syslog() is a more modern transport based on RFC6587 "framing", which defaults to RFC5424 parsing, and falls back to RFC3164 in case of failures.

The same could be done with the network() source, which might introduce some breaking behavior in special cases, but under normal circumstances, the fallback should work as before.

[MrAnno]

Nice-to-have.

[gaborznagy]

Disable truncating of disk-buffers (proposed by @alltilla ).

[MrAnno]

When disabled, pre-allocate full disk buffer size .
We keep the option to re-enable it.

Nice-to-have.

[alltilla]

#4056

[amdrsantos]

Log rotation based on size:
With a configurable number of history files.
With optional compression.

[MrAnno]

Thank you for the idea.

FYI a native size-based log rotation mechanism is one of our Google Summer of Code projects that we are working on with a student this year.

[MrAnno]

The progress can be followed in #4145.

[MrAnno]

Fixing template escaping:
#3158 (comment)

[MrAnno]

In progress.

[MrAnno]

• exclude $1, $2, $3 matches from value-pairs' --scope all-nv-pairs.
• nice-to-have: value-pairs improvements

[alltilla]

@OverOrion agreed to implement this.

[bazsi]

Do we need int64/int32 types?

[bazsi]

we don't need separate types, get rid off one of them and make "int" generic.

[MrAnno]

• we leave the int32 validator function intact in case it is needed in a destination, but internally, we will only have one single integer type.

[bazsi]

This is implemented in PR #4113

[Kokan]

drop threaded(no) support
this option only controls work thread pool, but many features of syslog-ng such as threaded destination/source (like http, kafka, ...) does deploy threads regardless. or just simple internal log, syslog-ng-ctl stats.

[alltilla]

@Kokan Do you plan on working on this?

[Kokan]

regex filter, there is already a regex parser. the parser should be used instead and no filter should be able to change the log message (either drop or accept it)

[bazsi]

I don't think we can change that now. A lot of configurations would use the match() -style filtering with flags(store-matches) which we would break.

Also, I am thinking of adding a perl-like =~ operator to our set of supported comparisons which would apply a regex and also set matches.

[alltilla]

We decided not to remove match() filter with store-matches.

[Kokan]

Normalize parsers behavior. Currently the parsers are not conforming, some drops the message if parsing was not successful some not. The prefix option either adds . or not.

                        drops?
add_contextual_data     no(should it be a parser?)
app-parser              no
csv-parser              yes
date-parser             yes
db-parser               yes
geoip2                  no
grouping-by             yes (but not based on 'parsing' result)
json-parser             no
kv-parser               no
linux-audit-parser      no(~kv-scanner)
map_value_pairs         no
python                  depends(on user code)
regexp-parser           yes
snmptrapd-parser        yes
syslog-parser           yes
tags-parser             no
xml                     yes(can be configured what to do)

[bazsi]

• add-contextual-data never fails, we could add an option to drop unmatched messages.
• App-parser drops, if nothing matches.
• Json-parser drops if parsing fails
• Kv-parser is a combination of a lot of heuristics, the whole point is to extract as much data as possible. We might want to create a stricter version that has no heuristics and drops in case of a parser error
• Tags-parser should drop in case if a parse error as it reverses $TAGS

[alltilla]

At least we should let the user configure what happens with each filter/parser. This is not a breaking change, so no need to do it for 4.0.

[Kokan]

The prefix part for sure a breaking change, and adding a configure option but changing the default behavior is a breaking change.

[bazsi]

can you point me to a parser where prefix automatically adds the last dot?

[gaborznagy]

I think @Kokan meant the xml-parser and geoip2. Although in xml-parser the default prefix is .xml, nevertheless each parsed child element in the xml is added with a separating dot character, e.g. .xml.Event.Data.

[bazsi]

Drop python2 support?

[Kokan]

https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-3.31.2

[alltilla]

#4057

[faxm0dem]

Add unit-test-like functionality to config (like patterndb examples)

[bazsi]

How to compare structured (JSON, list) name-value pairs? Right now we planned to compare these as strings, while escaping and whitespace could cause mismatches even if they are semantically the same.

[bazsi]

We've decided to keep the comparison as strings and should the use-case emerge to do better comparisons for these, we would take the template function path, e.g. add template functions that are specific to JSON, similarly to how we have functions for the list datatype.

[bazsi]

Exposing JSON name-value pairs to Python bindings should be done so they are not simply represented as str instances. There are two alternative ideas at the moment:

1. parse the JSON payload before returning it to Python using json.loads() function and then expose the resulting dict to the Python object. Do the reverse when a dict is assigned to a name-value pair (e.g. call json.dumps() and store that as a result).

2. create a new class that derives from str and allow Python code to do the parsing on its own.

[amdrsantos]

Support for syslog signed messages as in RFC 5848

[MrAnno]

Hi @amdrsantos,
The scope of syslog-ng v4 release has been established, its main topic will be all around message field types.

Please open a separate feature request, so we don't forget about it:
https://github.com/syslog-ng/syslog-ng/issues/new?assignees=&labels=enhancement&template=30-feature-request.md&title=