High operator memory usage during start-up

[swar8080]

Component(s)

Operator

What happened?

Description

Based on the discussion in this slack thread: https://cloud-native.slack.com/archives/C033BJ8BASU/p1712158076121409

We are seeing the OTEL operator consume up to 1.4gb of memory during start-up before settling to ~600mb. This is a cluster with ~7k pods, about a dozen OTEL collectors, and so far only a few pods using Instrumentation for auto-injection. This happened recently after bumping the java instrumentation version to inject from 2.0.0 to 2.2.0. No issues in our other (smaller) clusters.

We raised the k8 memory limit to 3gb for now

Configuration

        - repoURL: "https://open-telemetry.github.io/opentelemetry-helm-charts"
          targetRevision: "0.44.2"
          chart: opentelemetry-operator
          helm:
            valuesObject:
              admissionWebhooks:
                certManager:
                  create: true
              manager:
                podLabels:
                  grafana.agent.log.collection: monit
                collectorImage:
                  repository: our.custom.artifactory:6555/otelcontrib-our-fork
                  tag: // Omitted 
                autoInstrumentationImage:
                  java:
                    repository: "ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java"
                    tag: "2.2.0"

Kubernetes Version

1.24

Operator version

0.90.0

Collector version

A minor fork of 0.94.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

2024-04-03 17:37:12.648	
I0403 17:37:12.648170       1 leaderelection.go:260] successfully acquired lease observability/9f7554c3.opentelemetry.io
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","logger":"collector-upgrade","msg":"looking for managed instances to upgrade"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1alpha1.OpenTelemetryCollector"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.ConfigMap"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.ServiceAccount"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","logger":"instrumentation-upgrade","msg":"looking for managed Instrumentation instances to upgrade"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.Service"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.Deployment"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.DaemonSet"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.StatefulSet"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v2.HorizontalPodAutoscaler"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","source":"kind source: *v1.PodDisruptionBudget"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting Controller","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1alpha1.OpAMPBridge"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.ConfigMap"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.ServiceAccount"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.Service"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting EventSource","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","source":"kind source: *v1.Deployment"}
2024-04-03 17:37:12.648	
{"level":"info","ts":"2024-04-03T17:37:12Z","msg":"Starting Controller","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge"}
2024-04-03 17:37:12.754	
{"level":"info","ts":"2024-04-03T17:37:12Z","logger":"collector-upgrade","msg":"no instances to upgrade"}
2024-04-03 17:37:12.760	
{"level":"error","ts":"2024-04-03T17:37:12Z","logger":"instrumentation-upgrade","msg":"autoinstrumentation not enabled for this language","flag":"operator.autoinstrumentation.go","stacktrace":"github.com/open-telemetry/opentelemetry-operator/pkg/instrumentation/upgrade.(*InstrumentationUpgrade).upgrade\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/pkg/instrumentation/upgrade/upgrade.go:135\ngithub.com/open-telemetry/opentelemetry-operator/pkg/instrumentation/upgrade.(*InstrumentationUpgrade).ManagedInstances\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/pkg/instrumentation/upgrade/upgrade.go:75\nmain.addDependencies.func2\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/main.go:345\nsigs.k8s.io/controller-runtime/pkg/manager.RunnableFunc.Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/manager.go:301\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/runnable_group.go:223"}
2024-04-03 17:37:12.760	
{"level":"error","ts":"2024-04-03T17:37:12Z","logger":"instrumentation-upgrade","msg":"autoinstrumentation not enabled for this language","flag":"operator.autoinstrumentation.nginx","stacktrace":"github.com/open-telemetry/opentelemetry-operator/pkg/instrumentation/upgrade.(*InstrumentationUpgrade).upgrade\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/pkg/instrumentation/upgrade/upgrade.go:135\ngithub.com/open-telemetry/opentelemetry-operator/pkg/instrumentation/upgrade.(*InstrumentationUpgrade).ManagedInstances\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/pkg/instrumentation/upgrade/upgrade.go:75\nmain.addDependencies.func2\n\t/home/runner/work/opentelemetry-operator/opentelemetry-operator/main.go:345\nsigs.k8s.io/controller-runtime/pkg/manager.RunnableFunc.Start\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/manager.go:301\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/manager/runnable_group.go:223"}
2024-04-03 17:37:20.263	
{"level":"info","ts":"2024-04-03T17:37:20Z","msg":"Starting workers","controller":"opampbridge","controllerGroup":"opentelemetry.io","controllerKind":"OpAMPBridge","worker count":1}
2024-04-03 17:37:20.358	
{"level":"info","ts":"2024-04-03T17:37:20Z","msg":"Starting workers","controller":"opentelemetrycollector","controllerGroup":"opentelemetry.io","controllerKind":"OpenTelemetryCollector","worker count":1}
2024-04-03 17:37:20.428	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:20.428	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:37:20.489	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:20.489	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:37:20.551	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:20.551	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:37:20.621	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:20.621	
{"level":"info","ts":"2024-04-03T17:37:20Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:37:21.403	
{"level":"info","ts":"2024-04-03T17:37:21Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:21.403	
{"level":"info","ts":"2024-04-03T17:37:21Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:37:22.122	
{"level":"info","ts":"2024-04-03T17:37:22Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:22.122	
{"level":"info","ts":"2024-04-03T17:37:22Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:37:24.194	
{"level":"info","ts":"2024-04-03T17:37:24Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:24.194	
{"level":"info","ts":"2024-04-03T17:37:24Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:37:36.432	
{"level":"info","ts":"2024-04-03T17:37:36Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:36.432	
{"level":"info","ts":"2024-04-03T17:37:36Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:37:37.135	
{"level":"info","ts":"2024-04-03T17:37:37Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:37.135	
{"level":"info","ts":"2024-04-03T17:37:37Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:37:39.221	
{"level":"info","ts":"2024-04-03T17:37:39Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:39.222	
{"level":"info","ts":"2024-04-03T17:37:39Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:37:51.453	
{"level":"info","ts":"2024-04-03T17:37:51Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:51.453	
{"level":"info","ts":"2024-04-03T17:37:51Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:37:52.147	
{"level":"info","ts":"2024-04-03T17:37:52Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:52.147	
{"level":"info","ts":"2024-04-03T17:37:52Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:37:54.288	
{"level":"info","ts":"2024-04-03T17:37:54Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:37:54.288	
{"level":"info","ts":"2024-04-03T17:37:54Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:38:06.467	
{"level":"info","ts":"2024-04-03T17:38:06Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:06.467	
{"level":"info","ts":"2024-04-03T17:38:06Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:38:07.167	
{"level":"info","ts":"2024-04-03T17:38:07Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:07.167	
{"level":"info","ts":"2024-04-03T17:38:07Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:38:09.313	
{"level":"info","ts":"2024-04-03T17:38:09Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:09.313	
{"level":"info","ts":"2024-04-03T17:38:09Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:38:21.494	
{"level":"info","ts":"2024-04-03T17:38:21Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:21.494	
{"level":"info","ts":"2024-04-03T17:38:21Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:38:22.184	
{"level":"info","ts":"2024-04-03T17:38:22Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:22.184	
{"level":"info","ts":"2024-04-03T17:38:22Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:38:24.323	
{"level":"info","ts":"2024-04-03T17:38:24Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:24.323	
{"level":"info","ts":"2024-04-03T17:38:24Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:38:36.519	
{"level":"info","ts":"2024-04-03T17:38:36Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:36.519	
{"level":"info","ts":"2024-04-03T17:38:36Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:38:37.198	
{"level":"info","ts":"2024-04-03T17:38:37Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:37.198	
{"level":"info","ts":"2024-04-03T17:38:37Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:38:39.338	
{"level":"info","ts":"2024-04-03T17:38:39Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:39.338	
{"level":"info","ts":"2024-04-03T17:38:39Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:38:51.543	
{"level":"info","ts":"2024-04-03T17:38:51Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:51.543	
{"level":"info","ts":"2024-04-03T17:38:51Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:38:52.217	
{"level":"info","ts":"2024-04-03T17:38:52Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:52.218	
{"level":"info","ts":"2024-04-03T17:38:52Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:38:54.358	
{"level":"info","ts":"2024-04-03T17:38:54Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:38:54.358	
{"level":"info","ts":"2024-04-03T17:38:54Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:39:06.568	
{"level":"info","ts":"2024-04-03T17:39:06Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:06.568	
{"level":"info","ts":"2024-04-03T17:39:06Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:39:07.230	
{"level":"info","ts":"2024-04-03T17:39:07Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:07.230	
{"level":"info","ts":"2024-04-03T17:39:07Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:39:09.365	
{"level":"info","ts":"2024-04-03T17:39:09Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:09.365	
{"level":"info","ts":"2024-04-03T17:39:09Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:39:21.584	
{"level":"info","ts":"2024-04-03T17:39:21Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:21.584	
{"level":"info","ts":"2024-04-03T17:39:21Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:39:22.247	
{"level":"info","ts":"2024-04-03T17:39:22Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:22.247	
{"level":"info","ts":"2024-04-03T17:39:22Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:39:24.379	
{"level":"info","ts":"2024-04-03T17:39:24Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:24.379	
{"level":"info","ts":"2024-04-03T17:39:24Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:39:36.616	
{"level":"info","ts":"2024-04-03T17:39:36Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:36.616	
{"level":"info","ts":"2024-04-03T17:39:36Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:39:37.262	
{"level":"info","ts":"2024-04-03T17:39:37Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:37.262	
{"level":"info","ts":"2024-04-03T17:39:37Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:39:39.414	
{"level":"info","ts":"2024-04-03T17:39:39Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:39.414	
{"level":"info","ts":"2024-04-03T17:39:39Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:39:51.628	
{"level":"info","ts":"2024-04-03T17:39:51Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:51.628	
{"level":"info","ts":"2024-04-03T17:39:51Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}
2024-04-03 17:39:52.280	
{"level":"info","ts":"2024-04-03T17:39:52Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-backends","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:52.280	
{"level":"info","ts":"2024-04-03T17:39:52Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-backends","namespace":"observability"}
2024-04-03 17:39:54.410	
{"level":"info","ts":"2024-04-03T17:39:54Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:39:54.410	
{"level":"info","ts":"2024-04-03T17:39:54Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-loadbalancer","namespace":"observability"}
2024-04-03 17:40:06.646	
{"level":"info","ts":"2024-04-03T17:40:06Z","logger":"controllers.OpenTelemetryCollector","msg":"no upgrade routines are needed for the OpenTelemetry instance","name":"monit-otel-collectors-forwarder","namespace":"observability","version":"0.90.1","latest":"0.61.0"}
2024-04-03 17:40:06.646	
{"level":"info","ts":"2024-04-03T17:40:06Z","logger":"controllers.OpenTelemetryCollector","msg":"skipping upgrade for OpenTelemetry Collector instance","name":"monit-otel-collectors-forwarder","namespace":"observability"}

Additional context

No response

[jaronoff97]

Thanks for the details here, I'm going to take some profiles in my clusters where I run the operator and see if anything is jumping out performance wise.

[pavolloffay]

Maybe we need to improve the caching of configmaps and secrets https://sdk.operatorframework.io/docs/best-practices/designing-lean-operators/

[jaronoff97]

@pavolloffay i think you're right!

[jaronoff97]

config from string is certainly not helping though, I think we should look into getting #2735 which would help a lot here

[Rohlik]

We hit on the same issue on our EKS with ~86 nodes when the manager container within the opentelemetry-operator is enormously memory-hungry during its start, leading to OOMKills.
We are on the v0.107.0 version.