Mitigate prometheus RAM flooding

[cbeneke]

What this PR does / why we need it:

kubernetes/kubernetes#68530 fixes a bug in kubernetes, which floods the metrics created by the controller-manager. As the fix seems not to be backported to v1.11 this change will drop all rest_* series from the controller-manager when any clusterversion v1.11.x is used.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Special notes for your reviewer:

Documentation:

Release note:

Version v1.11.0 - 1.11.3 Clusters will no longer gather `rest_*` metrics from the controller-manager due to a [bug in kubernetes](https://github.com/kubernetes/kubernetes/pull/68530)

[cbeneke]

Hmm.. it seems I don't have access to the build log. I compared the Cluster.Version call to

kubermatic/api/pkg/resources/cloudconfig/configmap.go

Line 90 in 2157720

{{- if semverCompare ">=1.9.*" .Cluster.Spec.Version }}

.. updated to be in the TemplateData though, as I suspected the structure is comparable. Would be nice if someone can lighten me up how to correctly compare the version there. :)

[alvaroaleman]

I asked upstream if there will be a backport because IMHO there should be one: kubernetes/kubernetes#68530 (comment)

[cbeneke]

Thanks for fixing the PR :) If it gets cherry picked, the workaround should still be rolled out for versions v1.11.0 - v1.11.x (x being the release including the backport) for stability. (Or am I missing something?)

[alvaroaleman]

@cbeneke Yep exactly

[mrIncompetent]

The cherry pick for upstream has been merged.
We can now limit the affected versions to 1.11.0 - 1.11.3

[mrIncompetent]

@alvaroaleman @kron4eg @cbeneke
I updated the config to only drop those metrics for 1.11.0 - 1.11.3.
PTAL

[mrIncompetent]

/cherry-pick release/v2.8

[kubermatic-bot]

Error creating cherry-pick due to: error executing command git -C REDACTED cherry-pick 20aeed6f6677836021943400203e5709e221333d: out=error: could not apply 20aeed6f... Mitigate prometheus RAM flooding (#2020) hint: after resolving the conflicts, mark the corrected paths hint: with 'git add <paths>' or 'git rm <paths>' hint: and commit the result with 'git commit' , err=exit status 1