-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve syslog connection handling #4369
Conversation
Remove leading and trailing spaces and/or newlines from syslog message field.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@leodido Is timeout handling something we can push down to go-syslog? I think we can't handle this here unless we were to also handle the framing.
plugins/inputs/syslog/syslog.go
Outdated
return | ||
} | ||
// other error, log and return | ||
s.store(rfc5425.Result{Error: fmt.Errorf("failed reading from syslog client - %s", err.Error())}, acc) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just use acc.AddError(err) directly.
plugins/inputs/syslog/syslog.go
Outdated
} | ||
|
||
parseMsg: | ||
var p *rfc5425.Parser |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ideally we would only make one parser for each call to handle.
plugins/inputs/syslog/syslog.go
Outdated
p.ParseExecuting(func(r *rfc5425.Result) { | ||
s.store(*r, acc) | ||
}) | ||
n, err := io.Copy(data, conn) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Won't this read until EOF? How many messages can syslog send over a single connection. If it allows persistent connection then this could be very large.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A receiver MUST use the message length to delimit a syslog messages, not EOF
.
Ie., 14 octets for the following message <1>1 - - - - -
, thus 14 <1>1 - - - - -
.
RFC5425 specifies only that receivers SHOULD be able to process messages with lengths up to and including 8192 octets.
There is no maximum.
@danielnelson no we cannot push down to go-syslog imho, RFC5425 says nothing about timeout. |
Anyway the |
Without duplicating the syslog tcp framing, we can't do the message splitting in Telegraf since we don't know where messages begin or ends. I think this means we have to send in the connection io.Reader. So, I think we could handle the error in the ParseExecuting function ( However, what I think would be ideal is if go-syslog could return errors when returned from the Read function of the io.Reader provided. The application could then check the error and handle it in an way depending on the type of the io.Reader. |
(cherry picked from commit 69d22af)
This change reads from a syslog client in a separate step from the log parsing. This allows fewer errors in telegraf's logs about exceeding read timeouts. This should also limit the amount of reconnecting syslog clients must do when streaming logs to telegraf.
Previously, the syslog plugin would immediately start reading from a connection and if data was not immediately provided (or a connection was made and closed),
found EOF, expecting a MSGLEN
would be printed to telgraf's log.By reading the data on the connection in a separate step, we are able to handle connection issues separately from parsing issues, thus resulting in less of them. This will also allow for syslog client connections to stay alive longer.
Resolves #4335