chore(kuma-cp) disable log sampling

[jakubdyszkiewicz]

Summary

While stress testing KDS I noticed that some logs about creating resources are missing.
After more investigation, it turned out that even with a simple loop

for i := 0; i < 200; i++ {
  core.Log.Info("test", "x", i)
}

we are dropping logs after 100.

It's because we have log sampling enabled.

Why sample application logs?
Applications often experience runs of errors, either because of a bug or because of a misbehaving user. Logging errors is usually a good idea, but it can easily make this bad situation worse: not only is your application coping with a flood of errors, it's also spending extra CPU cycles and I/O logging those errors. Since writes are typically serialized, logging limits throughput when you need it most.

Sampling fixes this problem by dropping repetitive log entries. Under normal conditions, your application writes out every entry. When similar entries are logged hundreds or thousands of times each second, though, zap begins dropping duplicates to preserve throughput.

https://github.com/uber-go/zap/blob/master/FAQ.md#why-sample-application-logs

While I understand the rationale, I don't think it's the right solution - at least for Kuma and INFO logs. Maybe it's fine for errors with the same msg and key-value pairs, but I'd do this only if we have such a problem. Overall, It only makes debugging even more confusing.

If we have a loop that spams logs, we should introduce backoff, not drop logs.

Documentation

• No docs

Testing

• Unit tests
• E2E tests
• Manual testing on Universal
• Manual testing on Kubernetes

[bartsmykla]

lgtm