promtail/targets/syslog: Enable best effort parsing for Syslog messages

[ldb]

What this PR does / why we need it:
This change enables best effort parsing for Syslog messages in the Syslog target within Promtail.
Right now, a malformed message will stop the syslog parser. However, the criteria for a message to be considered "valid" is very low.
By using the best effort option, a malformed or partially parsed message will not stop the parsing loop anymore, which otherwise would eventually close the TCP connection.
Parsing errors will still be logged and tracked in metrics.
Malformed messages will not be passed back to Promtail.

Which issue(s) this PR fixes:
Fixes #5395
Fixes #4270

Also presumably:
Fixes #4871
Fixes #4808

Special notes for your reviewer:

Just checked this in our setup and was able to verify that this change indeed fixes TCP connection problems in Rsyslog (with error -2027).

Checklist

• Documentation added
• Tests updated
• Add an entry in the CHANGELOG.md about the changes.

[stephenafamo]

I really need this merged. The current syslog parser is extremely picky.

Thanks for this @ldb

[ldb]

@stephenafamo just to be clear, this PR does not change the parser to accept malformed messages. It just makes it so that a malformed Syslog message does not close the underlying transport anymore. Malformed messages will still not be passed to Loki in this way.

[stephenafamo]

I would try to test this by myself, but my issue was that the syslog messages generated by the log/syslog package was rejected by promtail.

1. log/syslog does not include the version
2. promtail seems to require every field including does that are allowed to be nil according to RFC5424. So even if you fork the package add the version (which I did), promtail still complains that procid, msgid, structured-data e.t.c. are missing even if they are not required to be present.

I am not sure I have an easy way to test your changes, but I'd like to see if the best-effort parsing is more usable -- at least with log/syslog.

[ldb]

There is no mention about log/syslog being compliant with RFC5424, the Syslog standard.
Looking at the code, it seems to be one of the earliest packages in the stdlib and actually has been frozen some while ago. Your best bet would likely be migrating away from it.

[stephenafamo]

Sadly I couldn't find a maintained alternative. Any suggestions would be welcome.

[slim-bean]

This LGTM and seems low risk, thanks for the contribution!

[ldb]

@slim-bean I did not add this to the CHANGELOG.md yet. Is this something I should update this branch for and make a new PR or what would be the best way?

[stephenafamo]

Update: Tried with these changes, still rejects the logs from the standard library's log/syslog package. This doesn't seem to be promtail's fault at all.