Rate limit duplicate log lines

[phemmer]

Feature Request

Opening a feature request kicks off a discussion.

Proposal:

Telegraf should rate limit duplicate log entries to prevent the same line from spamming the logs.

Current behavior:

There is no rate limiting on duplicate lines, so telegraf can repeat the same error hundreds of times per second.

Desired behavior:

Telegraf should keep track of what it has logged, and prevent the same log message from being output within an X-second period, where X is configurable.
Note that there should not be a restriction on the duplicate lines being consecutive to be suppressed. Meaning the following should not happen:

00:00:01 Error: foo
00:00:01 Error: bar
00:00:01 Error: foo
00:00:01 Error: bar

Use case: [Why is this important (helps with prioritizing requests)]

When errors occur in some of the plugins, they can be extremely repetitive in their messages. Sometimes generating hundreds of the same messages per second. This can drown out other more quiet errors. It can also result in log destinations being saturated, or filled up (e.g. bandwidth or disk space).

[phemmer]

One of the basic problems with solving this, and other things (such as tracking error count per-plugin, or starting telegraf in a new namespace via exec (#2087)), is that some of the plugins write errors directly to STDOUT/STDERR.

We could do something like opening a pipe and redirecting STDOUT/STDERR into the pipe, but with this we don't know which plugin generated the error, and it prevents us from properly tracking errors per plugin (#1348).

We could also force plugins to use Accumulator.AddError() by redirect STDOUT/STDERR to /dev/null. The only downside to this that I can think of is if a plugin wants to log non-error messages.

The only other option I can think of is to provide a per-plugin logger for plugins to use. We would again redirect STDOUT/STDERR to /dev/null.

I think these latter 2 options are the better solutions, as aside from addressing #1348 & #2087, it also makes it much easier for admins to ensure telegraf logs go to syslog. Personally this isn't a problem for me as my systems use systemd, so STDOUT/STDERR is automatically collected into the journal. But for non-systemd users, getting the logs into syslog is a lot harder/messier.

@sparrc thoughts?

[sparrc]

I think I'd prefer not to have a per-plugin logger, but could be convinced otherwise if it's really needed. Whatever we do, it needs to reflect influxdata/influxdb#7671

[phemmer]

So what does closing this mean? The issues mentioned will not be addressed?

[sparrc]

oops, didn't mean to close it

[MyaLongmire]

I believe this is handled by systemd and journalctl. Therefore this does not involve telegraf and I will close the issue. If you are still having problems feel free to open another issue.

[phemmer]

No, systemd does not handle this. Systemd will rate limit all log lines, not duplicate log lines. In fact this functionality is needed to prevent the systemd rate limit from suppressing important information.

But the primary importance is to prevent spammy plugins from drowning out legitimate messages