Is there a way with the OpenTelemetry Operator Auto-Instrumentation to instrument multiple languages in the same Pod

[jack78901]

I have an application running in Kubernetes that has a pod that has multiple containers. Each container performs part of a task. Namely, one is the API (written in node.js), a backend transport (written in java), and a drive connector (written in C).

When attempting to add multiple annotations for instrumentation, we get the error:

44s Warning FailedCreate statefulset/my-pod-with-multiple-containers create Pod my-pod-with-multiple-containers-1 in StatefulSet my-pod-with-multiple-containers failed error: Pod “my-pod-with-multiple-containers-1” is invalid: spec.containers[1].volumeMounts[2].mountPath: Invalid value: "/otel-auto-instrumentation": must be unique

I know that with OpenTelemetry, you can annotate a pod to do auto-instrumentation, such as:

apiVersion: apps/v1
kind: Statefulset
metadata:
name: my-deployment-with-multiple-containers
spec:
selector:
matchLabels:
app: my-pod-with-multiple-containers
replicas: 1
template:
metadata:
labels:
app: my-pod-with-multiple-containers
annotations:
instrumentation.opentelemetry.io/inject-java: "true"
instrumentation.opentelemetry.io/container-names: "myapp,myapp2"
spec:
containers:
- name: myapp
image: myImage1
- name: myapp2
image: myImage2
- name: myapp3
image: myImage3
The above (taken from https://github.com/open-telemetry/opentelemetry-operator/blob/main/README.md#multi-container-pods), myapp and myapp2 would get java auto-instrumentation injected.

We tried doing:

  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    instrumentation.opentelemetry.io/container-names: "myapp"
    instrumentation.opentelemetry.io/inject-nodejs: "true" 
    instrumentation.opentelemetry.io/container-names: "myapp2"
spec:
  containers:
  - name: myapp
    image: myImage1
  - name: myapp2
    image: myImage2
  - name: myapp3
    image: myImage3

We were hoping this would make it so that myapp would get Java instrumentation and myapp2 would get node.js instrumentation.

I was informed to move this here from Stack Overflow: https://stackoverflow.com/questions/75943971/is-there-a-way-with-the-opentelemetry-operator-auto-instrumentation-to-instrumen

[TylerHelmuth]

Unfortunately the Instrumentation resource cannot do what you need, but I think it is a great use case we should allow. Feels like the Instrumentation should be able to handle language-specific container names like instrumentation.opentelemetry.io/java-container-names that are the containers that should have the specified language injected.

[TylerHelmuth]

I started work on this today but ran into a snag with the initContainers. We restrict ourselves to 1 right now, which isn't a strict requirement, but before I dive down that rabbit hole:

@jack78901 have you tried adding the auto-instrumentation to your pod manually? Instead of using the operator, have you tried setting up your deployment so it has java initConatiners and nodejs initContainers and all the proper volume mounts and env vars? I want to confirm that what we're trying to do here is even possible with the way the different language auto-instrumentation work with initContainers.

[frzifus]

@TylerHelmuth I had a similar issue in the past. As far as I can remember, sdkInjector.inject in instPodMutator.Mutate cannot be executed twice. This is caused by a few append calls that do not check for an existing entry.

E.g.:

opentelemetry-operator/pkg/instrumentation/javaagent.go

Lines 55 to 58 in 87d4e10

	container.VolumeMounts = append(container.VolumeMounts, corev1.VolumeMount{
	Name: volumeName,
	MountPath: "/otel-auto-instrumentation",
	})

Once this method can be executed multiple times and leads to the same result we can also continue with #553 😮‍💨

[TylerHelmuth]

@frzifus I think we can definitely handle that blocker, especially if we gave each language its own volume mount and path. I'm more concerned tho that the languages will interfere with each other if we try to auto-instrument multiple languages. I'll try it out myself soon.

[jack78901]

@TylerHelmuth, I apologize. I missed in your message about testing the manual configuration to see if the multiple instrumentation will work. I will see if we can try this in our dev environment and see what results.

Thank you for looking into this!

[mat-rumian]

Hey @jack78901 @TylerHelmuth ,
I've started to work on instrumentation support for multiple containers. Code is almost ready, I'm testing solution.

[jaronoff97]

Closed by #1901, please let me know if that solution is not sufficient!