Excessive log messages

[lukehoban]

We've seen a few cases where a seemingly excessive number of log messages are generated by the kubernetes package.

In one recent case, it appears that the package was leading to ~200 log messages per second.

I suspect this is some await logic which can trigger a tight loop of feedback under some condition.

This appears to have been introduced around about the time we did a significant refactoring on the await logic (the first mention of this problem was soon after that) - though it may be unrelated.

I do not have a repro myself - but I believe users have hit this when running the https://github.com/pulumi/examples/tree/master/kubernetes-ts-exposed-deployment example (though possibly only when some error condition is triggered).

[hausdorff]

We stream a "status" message every time we observe a Kubernetes watch event, which could be arbitrarily many times -- basically any time the API server changes something and reports to us. This would be a consequence of switching from the polling strategy we previously had.

Even if this isn't the problem, we should probably throttle event reporting and report as batch every second or two.

[lukehoban]

Even if this isn't the problem, we should probably throttle event reporting and report as batch every second or two.

Indeed - though we also shouldn't spit out 12,000 lines of messages per minute total :-).

[lblackstone]

@lukehoban @hausdorff Is batching important, or just preventing duplicate messages from logging to the service? I've got a WIP that sends batches once per second, but the bigger issue here was that most of the messages were duplicates triggered by k8s events.

[lukehoban]

Is batching important, or just preventing duplicate messages from logging to the service?

I'm not sure batching is critical (if it ends up with the same number of total lines of output). Re-sending the same status message as the last status message sent is (I believe) the core problem.

[hausdorff]

I actually think we should not batch messages. I think we should just de-dup them.