Generate RBAC rules when `config.extensions.k8s_observer` is used

[ChristianCiach]

Component(s)

collector

Is your feature request related to a problem? Please describe.

I am trying to enable annotation-based autodiscovery of the receiver_creator receiver:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otelcol-daemonset
spec:
  mode: daemonset
  config:
    extensions:
      k8s_observer:
        observe_pods: true
    receivers:
      receiver_creator/logs:
        watch_observers: [k8s_observer]
        discovery:
          enabled: true
    processors: {}
    exporters:
      debug:
        verbosity: basic
    service:
      pipelines:
        logs:
          receivers: [receiver_creator/logs]
          exporters: [debug]
      extensions: [k8s_observer]

This fails due to RBAC permissions of the collector:

W0616 14:31:14.920655       1 reflector.go:569] k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:otelcol-system:otelcol-daemonset-collector" cannot list resource "pods" in API group "" at the cluster scope
E0616 14:31:14.920706       1 reflector.go:166] "Unhandled Error" err="k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:otelcol-system:otelcol-daemonset-collector\" cannot list resource \"pods\" in API group \"\" at the cluster scope" logger="UnhandledError"

The operator has permissions to deploy ClusterRoleBindings and so on (deployed via helm with manager.createRbacPermissions=true), but it doesn't seem to even attempt creating the necessary RBAC for the collector.

Describe the solution you'd like

Looking at the source code, it looks like the operator is capable of dynamically creating the necessary RBAC rules for certain receivers and processors (k8sobjects, k8sattributes), but it doesn't seem to care about the k8s_observer extension.

The operator should probably treat the k8s_observer extension similarly to the k8sobjects receiver regarding the RBAC rules. Similar to the k8sobjects receiver, k8s_observer can be configured to listen to different kinds of objects (observe_pods, observe_services and so on), so the generated RBAC rules should take these flags into account.

Describe alternatives you've considered

No response

Additional context

Using Opentelemetry-Collector v0.127.0 deployed via helm with manager.createRbacPermissions=true

[ChristianCiach]

Workaround: Configure a dummy k8sobjects receiver to trick the operator into creating the necessary RBAC resources:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
spec:
  config:
    receivers:
      # Not really used. Only declared to auto-generate RBAC:
      # https://github.com/open-telemetry/opentelemetry-operator/issues/4113
      k8sobjects:
        objects:
          - name: pods
            mode: watch
    exporters:
      nop: {}
    service:
      pipelines:
        logs/rbac_hack:
          receivers: [k8sobjects]
          exporters: [nop]

"If it's stupid but it works, it's not stupid"!

[ChristianCiach]

This should probably be its own issue, but I'm on the run. So I post it here before I forget:

When the k8s_observer is used, the operator should inject K8S_NODE_NAME env variable, for the same reason it does it for the kubeletstats receiver:

opentelemetry-operator/internal/components/receivers/kubeletstats.go

Lines 35 to 40 in 5c86398

	func generateKubeletStatsEnvVars(_ logr.Logger, config kubeletStatsConfig) ([]corev1.EnvVar, error) {
	// The documentation mentions that the K8S_NODE_NAME environment variable is required when using the serviceAccount auth type.
	// Also, it mentions that it is a good idea to use it for the Read Only Endpoint. Added always to make it easier for users.
	// https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/kubeletstatsreceiver/README.md
	return []corev1.EnvVar{
	{Name: "K8S_NODE_NAME", ValueFrom: &corev1.EnvVarSource{FieldRef: &corev1.ObjectFieldSelector{FieldPath: "spec.nodeName"}}},

See https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/observer/k8sobserver/README.md#example-config

I've never seen the k8s_observer being used without the host filter, so the operator should probably set this environment variable unconditionally.