inject-sdk annotation doesn't play well with AWS CNI

[nerdingassendible]

Issue description

We are running AWS EKS cluster with AWS CNI enabled and security groups for pods. When i instrument a deployment with inject-sdk annotation they will usually not work - I suspect a race condition between AWS CNI and OTEL.

Annotations

        instrumentation.opentelemetry.io/inject-sdk: "xxxxx"
        instrumentation.opentelemetry.io/container-names: "xxxxx"

Error

Warning BranchENIAnnotationFailed 43s (x12 over 52s) vpc-resource-controller failed to annotate pod with branch ENI details: Pod "xxxx" is invalid: spec: Forbidden: pod updates may not change fields other than spec.containers[].image, spec.initContainers[].image, spec.activeDeadlineSeconds, spec.tolerations(only additions to existing tolerations) orspec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)

          ... // 5 identical elements                                                                                                                                     
          {Name: "OTEL_TRACES_SAMPLER", Value: "parentbased_traceidratio"},                                                                                               
          {Name: "OTEL_TRACES_SAMPLER_ARG", Value: "0.25"},                                                                                                               
+         {                                                                                                                                                               
+           Name: "OTEL_RESOURCE_ATTRIBUTES_POD_NAME",                                                                                                                    
+           ValueFrom: &core.EnvVarSource{                                                                                                                                
+             FieldRef: &core.ObjectFieldSelector{APIVersion: "v1", FieldPath: "metadata.name"},                                                                          
+           },                                                                                                                                                            
+         },                                                                                                                                                              
+         {                                                                                                                                                               
+           Name: "OTEL_RESOURCE_ATTRIBUTES_NODE_NAME",                                                                                                                   
+           ValueFrom: &core.EnvVarSource{                                                                                                                                
+             FieldRef: &core.ObjectFieldSelector{APIVersion: "v1", FieldPath: "spec.nodeName"},                                                                          
+           },                                                                                                                                                            
+         },                                                                                                                                                              
          {                                                                                                                                                               
            Name: "OTEL_RESOURCE_ATTRIBUTES",                                                                                                                             
            Value: strings.Join({                                                                                                                                         
              ... // 131 identical bytes                                                                                                                                  
              "TEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.name=$(OTEL_RESOURCE_",                                                                                         
              "ATTRIBUTES_POD_NAME),k8s.replicaset.name=xxxxx",                                                                                         
+             ",k8s.node.name=$(OTEL_RESOURCE_ATTRIBUTES_NODE_NAME),k8s.pod.nam",                                                                                         
+             "e=$(OTEL_RESOURCE_ATTRIBUTES_POD_NAME),k8s.pod.uid=xxxx",                                                                                                                                                                                                                  
            }, ""),                                                                                                                                                       
            ValueFrom: nil,                                                                                                                                               
          },                                                                                                                                                              
        },                                                                                                                                                                
        Resources:    {Limits: {s"cpu": {i: {...}, s: "1300m", Format: "DecimalSI"}, s"memory": {i: {...}, s: "1700Mi", Format: "BinarySI"}, s"vpc.amazonaws.com/pod-eni":
 {i: {...}, s: "1", Format: "DecimalSI"}}, Requests: {s"cpu": {i: {...}, s: "1300m", Format: "DecimalSI"}, s"memory": {i: {...}, s: "1700Mi", Format: "BinarySI"}, s"vpc.amazonaws.com/pod-eni": {i: {...}, s: "1", Format: "DecimalSI"}}},                                                                                                                                                                                                                                             
  }       

Reproduce

1. Create an EKS cluster with AWS CNI plugin
2. Install OTEL operator
3. Create a deployment with OTEL annotation # this works
4. Add SecurityGroupPolicy to the deployment
5. delete the pod # the new pod will error, repeat deletion couple of times if it doesnt

Note

If I remove the SecurityGroupPolicy or OTEL annotations - all works fine.

Issue looks similar to #1366