Improve syslog connection handling

[glinton]

This change reads from a syslog client in a separate step from the log parsing. This allows fewer errors in telegraf's logs about exceeding read timeouts. This should also limit the amount of reconnecting syslog clients must do when streaming logs to telegraf.

Previously, the syslog plugin would immediately start reading from a connection and if data was not immediately provided (or a connection was made and closed), found EOF, expecting a MSGLEN would be printed to telgraf's log.

By reading the data on the connection in a separate step, we are able to handle connection issues separately from parsing issues, thus resulting in less of them. This will also allow for syslog client connections to stay alive longer.

Resolves #4335

[danielnelson]

@leodido Is timeout handling something we can push down to go-syslog? I think we can't handle this here unless we were to also handle the framing.

[danielnelson]

Just use acc.AddError(err) directly.

[danielnelson]

Ideally we would only make one parser for each call to handle.

[danielnelson]

Won't this read until EOF? How many messages can syslog send over a single connection. If it allows persistent connection then this could be very large.

[leodido]

A receiver MUST use the message length to delimit a syslog messages, not EOF.
Ie., 14 octets for the following message <1>1 - - - - -, thus 14 <1>1 - - - - -.

RFC5425 specifies only that receivers SHOULD be able to process messages with lengths up to and including 8192 octets.

There is no maximum.

[leodido]

@danielnelson no we cannot push down to go-syslog imho, RFC5425 says nothing about timeout.

[leodido]

Anyway the found EOF, expecting MSGLEN error it the intended behavior (by RFC) at the moment for empty inputs, test here.

[danielnelson]

Without duplicating the syslog tcp framing, we can't do the message splitting in Telegraf since we don't know where messages begin or ends. I think this means we have to send in the connection io.Reader.

So, I think we could handle the error in the ParseExecuting function (Syslog.store) by watching for the found EOF, expecting MSGLEN message. The downside of this method is we don't know if the connection was closed, a timeout occured or something else. This makes good error messages hard to produce. Also, we would need to match on the error string which is not preferred. As an aside, it also seems like we need to extend the deadline in this function to prevent errors every $timeout seconds.

However, what I think would be ideal is if go-syslog could return errors when returned from the Read function of the io.Reader provided. The application could then check the error and handle it in an way depending on the type of the io.Reader.