Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add syslog parser #3645

Merged
merged 1 commit into from Nov 22, 2023
Merged

Add syslog parser #3645

merged 1 commit into from Nov 22, 2023

Conversation

eliaskosunen
Copy link
Contributor

@eliaskosunen eliaskosunen commented Nov 15, 2023

This PR expands the syslog format already implemented, to add support for RFC 3164 -compliant(ish) syslogs, called legacy_syslog in the code. RFC 5424 support was already implemented.

Alongside that, a syslog plugin is added, which can be used to parse syslogs as a part of a pipeline.

# RFC 5424 (schema = syslog.rfc5424):
$ echo "<30>1 2020-03-02T19:44:50.224836+01:00 parallels-Parallels-Virtual-Platform packagekitd 1370 - -  Parent finished..."
  | tenzir 'from stdin read syslog'
{
  "facility": 3,
  "severity": 6,
  "version": 1,
  "ts": "2020-03-02T18:44:50.224835",
  "hostname": "parallels-Parallels-Virtual-Platform",
  "app_name": "packagekitd",
  "process_id": "1370",
  "message_id": "",
  "message": " Parent finished..."
}


# RFC 3164 (schema = syslog.rfc3164):
$ echo "Nov 16 14:55:56 mymachine/10.1.1.16 PROGRAM: Freeform message"
  | tenzir 'from stdin read syslog'
{
  "facility": null,
  "severity": null,
  "timestamp": "Nov 16 14:55:56",
  "host": "mymachine/10.1.1.16",
  "tag": "PROGRAM",
  "content": "Freeform message"
}

# Malformed (schema = syslog.unknown):
$ echo "foobar" | tenzir 'from stdin read syslog'
{
  "syslog_message": "foobar"
}

Possible extensions (to do here, or in a separate PR):

  • Specify RFC version as an argument (note, that 3164 and 5424 are never ambiguous, and can always be separated)
  • Opt-in to require well-formed input
  • Parse timestamp into an actual datetime (Probably opt-in, with an option to specify the year)

@eliaskosunen eliaskosunen added feature New functionality format Parser and printer labels Nov 15, 2023
@eliaskosunen eliaskosunen force-pushed the topic/syslog-parser branch 2 times, most recently from 735ebb4 to 4c8f248 Compare November 16, 2023 14:11
@eliaskosunen eliaskosunen marked this pull request as ready for review November 16, 2023 15:50
@eliaskosunen
Copy link
Contributor Author

I'll still need to add a changelog entry, and a new format to the docs.

Copy link
Contributor

@jachris jachris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for continuing this 🙏 Below are just some first reactions.

libtenzir/builtins/formats/syslog.cpp Outdated Show resolved Hide resolved
libtenzir/include/tenzir/format/syslog.hpp Show resolved Hide resolved
libtenzir/src/format/syslog.cpp Outdated Show resolved Hide resolved
tenzir/integration/tests.yaml Outdated Show resolved Hide resolved
web/docs/formats/syslog.md Outdated Show resolved Hide resolved
libtenzir/include/tenzir/format/syslog.hpp Outdated Show resolved Hide resolved
libtenzir/builtins/formats/syslog.cpp Outdated Show resolved Hide resolved
libtenzir/builtins/formats/syslog.cpp Outdated Show resolved Hide resolved
web/docs/formats/syslog.md Outdated Show resolved Hide resolved
@eliaskosunen eliaskosunen force-pushed the topic/syslog-parser branch 2 times, most recently from 3962bb2 to 9ea4bbc Compare November 21, 2023 09:19
@dominiklohmann dominiklohmann merged commit a168c73 into main Nov 22, 2023
40 of 42 checks passed
@dominiklohmann dominiklohmann deleted the topic/syslog-parser branch November 22, 2023 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New functionality format Parser and printer
Projects
None yet
4 participants