fix: improve memory usage by using metadata informer

[sthaha]

This patch uses metadatainformer package to watch secrets and configmaps so that all content need not be loaded into memory thus reducing the amount of memory consumed.

Description

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request.
If it fixes a bug or resolves a feature request, be sure to link to that issue.

Type of change

What type of changes does your code introduce to the Prometheus operator? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Prometheus Operator now consumes less memory that before when there are lots of secrets and config-maps
present in the cluster.

- Optimised to consume less memory when there are lots of secrets or configmaps in the cluster. 

[simonpasquier]

lgtm. Do you have numbers comparing memory usage before/after the PR?
If this is useful, the same change should be done for the alertmanager and thanosruler controllers (but can be follow-up PRs).

[simonpasquier]

it doesn't read correctly?

[sthaha]

Good catch! :) sometimes, find and replace just doesn't cut it. 😂

[JoaoBraveCoding]

lgtm! Awsome work, I wasn't aware that such informers existed 🤯

[sthaha]

lgtm! Awsome work, I wasn't aware that such informers existed 🤯

Thanks to @simonpasquier for pointing it out! 🙇

[sthaha]

lgtm. Do you have numbers comparing memory usage before/after the PR?

@simonpasquier yes, I have been updating #5410 with new numbers. I will update here as well. Thanks a lot for the feedback

[simonpasquier]

lgtm. Waiting for the latest numbers before merging :)
As I wrote earlier, we probably need the same change for Alertmanager & ThanosRuler but it can be a follow-up PR.

[sthaha]

Here is the difference between 2 versions after adding 100 secrets; tldr is ...

Current Operator :  Showing nodes accounting for 302.26MB, 97.87% of 308.83MB total
New Operator     :  Showing nodes accounting for 4860.48kB, 100% of 4860.48kB total

create-secrets.sh

#!/usr/bin/env bash

set -eu -o pipefail

main() {
	local ns=$1
	shift

	local num=$1
	shift

	dd if=/dev/urandom bs=756439 count=1 | base64 >tmp/random.txt
	kubectl create ns "$ns" || true
	for i in $(seq $num); do
		{
			kubectl delete -n "$ns" secret "foo-${i}" || true
			kubectl create -n "$ns" secret generic "foo-${i}" --from-file=key=tmp/random.txt
		} &
		while [[ $(jobs -r | wc -l) -gt 30 ]]; do wait -n; done
	done
	wait
	rm -f tmp/random.txt

}
main "$@"

./create-secrets.sh foobar 1000

New Operator using MetadataInformer

File: operator
Type: inuse_space
Time: Mar 22, 2023 at 9:44am (AEST)
Showing nodes accounting for 5695.21kB, 100% of 5695.21kB total
      flat  flat%   sum%        cum   cum%
 1057.98kB 18.58% 18.58%  1057.98kB 18.58%  regexp/syntax.(*compiler).inst (inline)
  528.17kB  9.27% 27.85%  1556.17kB 27.32%  runtime/pprof.writeHeapInternal
  520.04kB  9.13% 36.98%   520.04kB  9.13%  k8s.io/utils/buffer.NewRingGrowing (inline)
     514kB  9.03% 46.01%     1028kB 18.05%  runtime/pprof.(*profileBuilder).emitLocation
     514kB  9.03% 55.03%      514kB  9.03%  runtime/pprof.(*profileBuilder).stringIndex (inline)
  512.56kB  9.00% 64.03%   512.56kB  9.00%  github.com/prometheus/client_golang/prometheus.(*metricMap).getOrCreateMetricWithLabelValues
  512.19kB  8.99% 73.03%   512.19kB  8.99%  golang.org/x/net/http2.(*ClientConn).RoundTrip
  512.14kB  8.99% 82.02%   512.14kB  8.99%  k8s.io/api/apps/v1.init
  512.07kB  8.99% 91.01%   512.07kB  8.99%  github.com/aws/aws-sdk-go/aws/endpoints.init
  512.05kB  8.99%   100%   512.05kB  8.99%  regexp/syntax.(*parser).newRegexp (inline)
         0     0%   100%   512.05kB  8.99%  github.com/asaskevich/govalidator.init
         0     0%   100%   512.56kB  9.00%  github.com/prometheus-operator/prometheus-operator/pkg/client/informers/externalversions/monitoring/v1.NewFilteredThanosRulerInformer.func1
         0     0%   100%   512.56kB  9.00%  github.com/prometheus-operator/prometheus-operator/pkg/client/versioned/typed/monitoring/v1.(*thanosRulers).List
         0     0%   100%   512.56kB  9.00%  github.com/prometheus-operator/prometheus-operator/pkg/k8sutil.(*clientGoHTTPMetricAdapter).Observe
         0     0%   100%   512.19kB  8.99%  github.com/prometheus-operator/prometheus-operator/pkg/operator.(*instrumentedListerWatcher).Watch
         0     0%   100%   520.04kB  9.13%  github.com/prometheus-operator/prometheus-operator/pkg/prometheus/server.(*Operator).Run
         0     0%   100%   520.04kB  9.13%  github.com/prometheus-operator/prometheus-operator/pkg/prometheus/server.(*Operator).addHandlers
         0     0%   100%   512.56kB  9.00%  github.com/prometheus/client_golang/prometheus.(*MetricVec).GetMetricWithLabelValues
         0     0%   100%   512.56kB  9.00%  github.com/prometheus/client_golang/prometheus.(*SummaryVec).GetMetricWithLabelValues

The current operator

File: operator
Type: inuse_space
Time: Mar 22, 2023 at 9:45am (AEST)
Showing nodes accounting for 302.26MB, 97.87% of 308.83MB total
Dropped 67 nodes (cum <= 1.54MB)
      flat  flat%   sum%        cum   cum%
  293.73MB 95.11% 95.11%   294.75MB 95.44%  io.ReadAll
    6.02MB  1.95% 97.06%     6.02MB  1.95%  golang.org/x/net/http2.glob..func3
    2.50MB  0.81% 97.87%     2.50MB  0.81%  runtime.allocm
         0     0% 97.87%     6.52MB  2.11%  golang.org/x/net/http2.(*ClientConn).readLoop
         0     0% 97.87%     6.02MB  1.95%  golang.org/x/net/http2.(*clientConnReadLoop).processData
         0     0% 97.87%     6.52MB  2.11%  golang.org/x/net/http2.(*clientConnReadLoop).run
         0     0% 97.87%     6.02MB  1.95%  golang.org/x/net/http2.(*dataBuffer).Write
         0     0% 97.87%     6.02MB  1.95%  golang.org/x/net/http2.(*dataBuffer).lastChunkOrAlloc
         0     0% 97.87%     6.02MB  1.95%  golang.org/x/net/http2.(*pipe).Write
         0     0% 97.87%     6.02MB  1.95%  golang.org/x/net/http2.getDataBufferChunk
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/informers/core/v1.NewFilteredSecretInformer.func1
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/kubernetes/typed/core/v1.(*secrets).List
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/rest.(*Request).Do
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/rest.(*Request).Do.func1
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/rest.(*Request).request
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/rest.(*Request).request.func3
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/rest.(*Request).request.func3.1 (inline)
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/rest.(*Request).transformResponse
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/tools/cache.(*ListWatch).List
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/tools/cache.(*Reflector).list.func1
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/tools/cache.(*Reflector).list.func1.2
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/tools/pager.(*ListPager).List
         0     0% 97.87%   294.75MB 95.44%  k8s.io/client-go/tools/pager.SimplePageFunc.func1
         0     0% 97.87%     3.55MB  1.15%  runtime.doInit
         0     0% 97.87%     3.55MB  1.15%  runtime.main

[simonpasquier]

great!