PodGC default threshold causes OOMs on small master machines.

[wojtek-t]

I've seen a cluster (from 1.2.4) where there is a job and the pods it is starting are simply crashing.
The problem is that those failed pods are not being garbage collected.

In the example cluster I've seen, we were able to produce ~13.000 failed pods over the week and it seems it none of these were ever garbage collected.

This basically results in constantly increasing memory usage of master components.
This seems like a bug to me.

@soltysh @gmarek @erictune @kubernetes/goog-control-plane @lavalamp @roberthbailey

[wojtek-t]

To clarify - the cluster was from 1.2.4 release.

[gmarek]

To be clear. The problem is that GC kicks in only after accumulating 12.5k Pods in the system and small (1, 2 core) master machines can't really handle this amount of load.

[davidopp]

ref/ #22680

[lavalamp]

Yeah. There's a flag for this already.

[wojtek-t]

@lavalamp - why did you close that?
Even though the flag exists, it's not used and this is a real problem.

[lavalamp]

@wojtek-t because this is a dup-- we already have #22680 and #25831 filed about this.