Inject otelcol sidecar into any namespace

[pavolloffay]

Signed-off-by: Pavol Loffay p.loffay@gmail.com

Resolves #199

• This PR adds support for injecting otelcol sidecar into any namespace - currently the sidecar can be injected into workloads that are in the same namespace as the otelcol CR.
• The injection is enabled via the same annotation - sidecar.opentelemetry.io/inject: "default/simplest" (default is the namespace)
• The PR changes how the sidecar config is handled - now the collector config is passed an env var to to the collector container. ---- This approach was chosen to eliminate issues with cleaning the config maps needed for the collector.

Tested on with:

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: simplest
spec:
  mode: sidecar
  config: |
    receivers:
      jaeger:
        protocols:
          thrift_compact:
    processors:

    exporters:
      logging:

    service:
      pipelines:
        traces:
          receivers: [jaeger]
          processors: []
          exporters: [logging]
EOF

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: myapp
  annotations:
    sidecar.opentelemetry.io/inject: "default/simplest"
spec:
  containers:
  - name: myapp
    image: jaegertracing/vertx-create-span:operator-e2e-tests
    ports:
      - containerPort: 8080
        protocol: TCP
EOF

Result:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{"sidecar.opentelemetry.io/inject":"default/simplest"},"name":"myapp","namespace":"ploffay"},"spec":{"containers":[{"image":"jaegertracing/vertx-create-span:operator-e2e-tests","name":"myapp","ports":[{"containerPort":8080,"protocol":"TCP"}]}]}}
    sidecar.opentelemetry.io/inject: default/simplest
  creationTimestamp: "2023-02-01T10:54:42Z"
  labels:
    sidecar.opentelemetry.io/injected: default.simplest
  name: myapp
  namespace: ploffay
  resourceVersion: "6276"
  uid: 232aea7f-3630-4255-b259-43355a404013
spec:
  containers:
  - image: jaegertracing/vertx-create-span:operator-e2e-tests
    imagePullPolicy: IfNotPresent
    name: myapp
    ports:
    - containerPort: 8080
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hllwm
      readOnly: true
  - args:
    - --config=env:OTEL_CONFIG
    env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: OTEL_CONFIG
      value: |
        receivers:
          jaeger:
            protocols:
              thrift_compact:
        processors:

        exporters:
          logging:

        service:
          pipelines:
            traces:
              receivers: [jaeger]
              processors: []
              exporters: [logging]
    - name: OTEL_RESOURCE_ATTRIBUTES_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: OTEL_RESOURCE_ATTRIBUTES_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: OTEL_RESOURCE_ATTRIBUTES_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: k8s.namespace.name=ploffay,k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.pod.uid=$(OTEL_RESOURCE_ATTRIBUTES_POD_UID)
    image: ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector:0.68.0
    imagePullPolicy: IfNotPresent
    name: otc-container
    ports:
    - containerPort: 6831
      name: jaeger-thrift-c
      protocol: UDP
    - containerPort: 8888
      name: metrics
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hllwm
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: minikube
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-hllwm
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

[pavolloffay]

Changing the API bc the function uses only the otelcol instance

[pavolloffay]

@jaronoff97 @Aneurysm9 could you please take a look and give me 👍🏼 on the approach?

The approach was chosen to avoid copying otelcol config map to the namespace where the injection is enabled and as well avoid issues with cleaning the config map.

My notes:

• owner reference cannot reference objects from a different namespace
• when a pod is created directly it does not have any owner references

Once we agree I will make the init container image configurable via a flag on the operator (if needed expose it in the CR as well).

[Aneurysm9]

Can you expand on the concerns with using config maps? While the proposed system appears to work, it's got more moving parts than I like.

[frzifus]

Basically I have nothing against the initcontainer solution, but I would prefer a uniform solution for internal and external namespaces. Otherwise there is the possibility that the implementations drift apart.

[pavolloffay]

Can you expand on the concerns with using config maps?

Correctly implementing the config map approach for the sidecar is tricky:

• The sidecar config map would have to be created in the target namespace.
• A custom cleaning would have to be implemented. An owner reference to otelcol cannot be used (it's a cross namespace) nor owner reference to a pod (a pod UID is not known in the admission webhook).

The init container approach is a cleaner approach because no cleaning is required and the exact sidecar configuration is known (via the OTEL_CONFIG env var and cat config.yaml in the initContainer). In the current implementation in the main branch it is hard to figure out what is the already injected sidecar configuration if the otelcol CR changes.

but I would prefer a uniform solution for internal and external namespaces. Otherwise there is the possibility that the implementations drift apart.

The init container approach that this PR introduces is used for both internal and external namespaces. There are as well tests (unit+e2e) to ensure the solution works. The deployment/statefulset/daemonset deployment modes keep using the mounted configmap as before - which makes perfect sense bc there is only a single version of the "instance".

[Aneurysm9]

Discussed with @pavolloffay OOB and I recommend setting the configuration in an environment variable on the collector container and using the config from there by setting the argument --config=env:OTEL_CONFIG. This avoids the need for an init container simply to copy the config and leaves nothing additional to clean up.

[pavolloffay]

It was a great suggestion, I didn't know that --config=env:OTEL_CONFIG is supported.

I have changed the PR to use this approach without the init container.

[jaronoff97]

I just had two questions/concerns, otherwise I think this approach makes sense.