Prevent fluentd from handling records containing its own logs.

[vsinghal13]

Description

We do not want to ingest the demo log which we use for fluentd liveness probe to be ingested in Sumo. This PR prevents fluentd from handling records containing its own logs.

Testing performed

• ci/build.sh
• Redeploy fluentd and fluentd-events pods
• Confirm events, logs, and metrics are coming in

[samjsong]

Are these the new "health check" logs we've been seeing? LGTM

[rvmiller89]

This won't prevent us from ingesting the dropped chunk message?

[perk-sumo]

If we drop all the fluentd logs altogether we won't be able to troubleshoot fluentd problems.
Maybe it's better to drop only the liveness health check logs?

[perk-sumo]

My suggestion is: filter liveness logs out and add some throttling for the fluentd logs, there is a plugin for that: https://github.com/rubrikinc/fluent-plugin-throttle

[vsinghal13]

Are these the new "health check" logs we've been seeing? LGTM

Yes, these are the ones sent via the liveness probe.

[vsinghal13]

If we drop all the fluentd logs altogether we won't be able to troubleshoot fluentd problems.
Maybe it's better to drop only the liveness health check logs?

The fluentd logs for troubleshooting will also be written with container tags.