(r)syslog onramp / preprocessor

[Licenser]

Enable Tremor to receive and send Syslog Protocol Messages (https://tools.ietf.org/html/rfc5424) , supporting as much syslog implementations as possible that might deviate from the standard.

In the wild we have different syslog protocols being used, the standard IETF format and the old BSD format. So ideally we should support both.

Receiving Syslog Messages

via UDP

Syslog messages are usually sent via UDP where 1 UDP packet contains 1 syslog message. We already support receiving data via UDP with our UDP onramp. We need a way to turn the packet data we receive into a structured Value. For this, we have codecs. A syslog codec should be able to handle both syslog message formats, or we write two different codecs, one for each format.

via TCP/TLS

The story for supporting syslog over TLS/TCP is a bit more involved. We currently do not support TLS over our TCP onramp, so this needs to be added. This is a major milestone towards full syslog support.

Given we have TLS, in order to support syslog messages over TCP, we need to support the RFC 5425 transport, that contains a textual length prefix before each message. This could be handled with a Preprocessor similar to the length-prefixed preprocessor.

Sending Syslog Messages

via UDP

For sending syslog messages, we need to turn structured data in an Event (Value) into the the syslog protocol format.

via TCP/TLS

For sending messages over TCP/TLS, we also need to add the textual length-prefix used in RFC 5425. Tremor already supports sending data via UDP and TCP via offramps.

The TCP offramp needs to get TLS support for supporting sending syslog messages via TCP/TLS.

Checklist

Phase 1 - receive syslog via UDP

• receive syslog data via UDP (onramp) and turn syslog messages into structured events
• turn structured events into textual syslog messages and send them out via UDP (offramp)

Phase 2 - syslog via TCP/TLS

• add support for the RFC 5425 transport protocol (textual length prefix)

Phase 3 - TLS support for TCP

• add support for receiving TLS encrypted data via TCP onramp
• add support for sending TLS encrypted data via TCP offramp

Reference

• IETF Syslog Message Format RFC 5424
• BSD Syslog Message Format RFC 3164
• Syslog Transport via UDP RFC 5426
• Syslog Transport via TLS/TCP RFC 5425

[glitzflitz]

As this issue is open for lfx mentorship are there any tasks that need to be done during application period? I'm interested in working on this issue.

[Licenser]

Hi @glitzflitz thanks for your interest!

We don't require anything outside of the LFX-Mentorship process I'll share here what we shared on your discord that should answer all questions :) if not feel free to pop by we're happy to help anyone to go through the process.

Hello everyone interested in the LFX-Mentorship program. To prevent confusion we want to clarify the timeline and process for the program (the document can be found here: https://github.com/cncf/mentoring/blob/master/lfx-mentorship/2021/01-Spring/README.md#timeline)

* Mentee applications are open until February the 12th (this Friday) if you wish to participate please read the Mentee guide for this: https://docs.linuxfoundation.org/lfx/mentorship/mentee-guide
* After that, from February the 15th to the 26th the applications will be reviewed and a decision made on the mentee that is awarded the mentorship, along with some administrivia related to that.
* The mentorship, and work on the tickets will begin on March the 1st. Please do not work on the tickets prior to that, until the review period is over (Feb 26th) there can no guarantee be made on assignments.

In the meantime, feel free to ask questions. We'll update any questions to the tasks on the corresponding tickets to ensure visibility, and every updates on process to the #mentoring channel here.

[glitzflitz]

@Licenser Thanks!

[Licenser]

@murex971 tagging you here can you give a quick reply so we can assign you the issue :)

[murex971]

@Licenser sure :)

[Licenser]

Thank you! And done :) congratulations to being the proud owner of this issue now ;)

[mfelsche]

Thank you @murex971 for working on this!