Newline handling in syslog (encoded as 15 12) #1

Closed
troy opened this Issue Oct 7, 2011 · 2 comments

Comments

3 participants
@troy
Contributor

troy commented Oct 7, 2011

Hi Jesper,

We're trying to support NLog as a sender for our hosted logging service, and encountering a somewhat strange issue. Newlines show up as the string "1512" since the string is being encoded in ASCII (https://github.com/graffen/NLog.Targets.Syslog/blob/master/src/NLog.Targets.Syslog/NLog.Targets.Syslog.cs#L349) and 15 is the ASCII character code for CR, 12 for LF (in octal).

The syslog RFCs don't define a specific way to handle newlines (or even mandate that senders or receivers allow them), and implementations behave differently, so there's nothing inherently wrong with that.

However, it made me wonder whether you've tried messages containing newlines (and meant to support them), and if so, whether this was an intentional decision or just the way that Encoding.ASCII.GetBytes happened to encode it.

We want to make sure that our receiver expects that input and displays them as newlines. If 15, 12 is intentional, we'll plan around that.

BTW, let me know if you have any questions about the syslog protocol or mechanics. I noticed in your blog you asked for feedback last year, and we have a lot of experience in that area (github.com/papertrail/remote_syslog, https://github.com/papertrail/syslog_protocol, and others). Always happy to try to answer questions or help with testing.

And thanks for a neat target, too. Cheers,

Troy

@troy

This comment has been minimized.

Show comment
Hide comment
@troy

troy Oct 10, 2011

Contributor

Just an update with some additional info (thanks @eric). The reason that newlines render as ASCII codes instead of the decoded entities is that RFC 3164 does have an opinion about newlines; they aren't allowed.

Here's the relevant part. The gist is that with the ASCII encoding which the syslog target is using, only certain character codes are supported. This is purely academic until someone sees 1512 where a newline should be, and then suddenly it's not ;-)

I think the right behavior might be to split the message at a newline and generate separate messages, with something like:

string[] lines = body.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);

or:

string[] lines = body.Split(new string[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries);

Here's the RFC note.

4.1.3 MSG Part of a syslog Packet

   The MSG part will fill the remainder of the syslog packet.  This will
   usually contain some additional information of the process that
   generated the message, and then the text of the message.  There is no
   ending delimiter to this part.  The MSG part of the syslog packet
   MUST contain visible (printing) characters.  The code set
   traditionally and most often used has also been seven-bit ASCII in an
   eight-bit field like that used in the PRI and HEADER parts.  In this
   code set, the only allowable characters are the ABNF VCHAR values
   (%d33-126) and spaces (SP value %d32).  However, no indication of the
   code set used within the MSG is required, nor is it expected.  Other
   code sets MAY be used as long as the characters used in the MSG are
   exclusively visible characters and spaces similar to those described
   above.  The selection of a code set used in the MSG part SHOULD be
   made with thoughts of the intended receiver.  A message containing
   characters in a code set that cannot be viewed or understood by a
   recipient will yield no information of value to an operator or
   administrator looking at it.
Contributor

troy commented Oct 10, 2011

Just an update with some additional info (thanks @eric). The reason that newlines render as ASCII codes instead of the decoded entities is that RFC 3164 does have an opinion about newlines; they aren't allowed.

Here's the relevant part. The gist is that with the ASCII encoding which the syslog target is using, only certain character codes are supported. This is purely academic until someone sees 1512 where a newline should be, and then suddenly it's not ;-)

I think the right behavior might be to split the message at a newline and generate separate messages, with something like:

string[] lines = body.Split(new string[] { Environment.NewLine }, StringSplitOptions.None);

or:

string[] lines = body.Split(new string[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries);

Here's the RFC note.

4.1.3 MSG Part of a syslog Packet

   The MSG part will fill the remainder of the syslog packet.  This will
   usually contain some additional information of the process that
   generated the message, and then the text of the message.  There is no
   ending delimiter to this part.  The MSG part of the syslog packet
   MUST contain visible (printing) characters.  The code set
   traditionally and most often used has also been seven-bit ASCII in an
   eight-bit field like that used in the PRI and HEADER parts.  In this
   code set, the only allowable characters are the ABNF VCHAR values
   (%d33-126) and spaces (SP value %d32).  However, no indication of the
   code set used within the MSG is required, nor is it expected.  Other
   code sets MAY be used as long as the characters used in the MSG are
   exclusively visible characters and spaces similar to those described
   above.  The selection of a code set used in the MSG part SHOULD be
   made with thoughts of the intended receiver.  A message containing
   characters in a code set that cannot be viewed or understood by a
   recipient will yield no information of value to an operator or
   administrator looking at it.
@graffen

This comment has been minimized.

Show comment
Hide comment
@graffen

graffen Oct 10, 2011

Contributor

Hi Troy!

Thanks for your email :-)

On Oct 7, 2011, at 5:57 PM, Troy Davis wrote:

However, it made me wonder whether you've tried messages containing newlines (and meant to support them), and if so, whether this was an intentional decision or just the way that Encoding.ASCII.GetBytes happened to encode it.

Actually, I haven't tried messages with multiple lines. Maybe I should do so when I get the time (I just switched jobs, so everything's a mess right now ;-))

We want to make sure that our receiver expects that input and displays them as newlines. If 15, 12 is intentional, we'll plan around that.

It's nothing I've done anything to handle specifically - but if you find a solution feel free to fork and submit a pull request.

BTW, let me know if you have any questions about the syslog protocol or mechanics. I noticed in your blog you asked for feedback last year, and we have a lot of experience in that area (github.com/papertrail/remote_syslog, https://github.com/papertrail/syslog_protocol, and others). Always happy to try to answer questions or help with testing.

Thanks, I just got confused about some simple binary shifting, that's all. Should learn not to try to understand stuff when I'm tired :-)

And thanks for a neat target, too. Cheers,

You're welcome! It was really just an experiment but I'm glad someone other than me finds it useful!

/Jesper

Contributor

graffen commented Oct 10, 2011

Hi Troy!

Thanks for your email :-)

On Oct 7, 2011, at 5:57 PM, Troy Davis wrote:

However, it made me wonder whether you've tried messages containing newlines (and meant to support them), and if so, whether this was an intentional decision or just the way that Encoding.ASCII.GetBytes happened to encode it.

Actually, I haven't tried messages with multiple lines. Maybe I should do so when I get the time (I just switched jobs, so everything's a mess right now ;-))

We want to make sure that our receiver expects that input and displays them as newlines. If 15, 12 is intentional, we'll plan around that.

It's nothing I've done anything to handle specifically - but if you find a solution feel free to fork and submit a pull request.

BTW, let me know if you have any questions about the syslog protocol or mechanics. I noticed in your blog you asked for feedback last year, and we have a lot of experience in that area (github.com/papertrail/remote_syslog, https://github.com/papertrail/syslog_protocol, and others). Always happy to try to answer questions or help with testing.

Thanks, I just got confused about some simple binary shifting, that's all. Should learn not to try to understand stuff when I'm tired :-)

And thanks for a neat target, too. Cheers,

You're welcome! It was really just an experiment but I'm glad someone other than me finds it useful!

/Jesper

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment