envFrom not working in ephemeral containers: failed to sync secret cache

[AdriaGS]

What happened?

I have the following deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mytarget
spec:
  selector:
    matchLabels:
      mylabel: mytarget
  replicas: 2
  template:
    metadata:
      labels:
        mylabel: mytarget
    spec:
      serviceAccountName: default
      containers:
        - name: mytarget
          image: <custom-image>
          command: ["sleep", "100d"]

I am working on an automated script (written in Go) that, among other things, does the following:

• Creates a Secret resource

s := apiv1.Secret{
	ObjectMeta: metav1.ObjectMeta{
		Name:      name,
		Namespace: metav1.NamespaceDefault,
	},
	StringData: map[string]string{
		"username": "foo",
		"password": "bar",
	},
}

_, err := clientset.CoreV1().Secrets(metav1.NamespaceDefault).Create(ctx, &s, metav1.CreateOptions{})

• Lists pods and creates one ephemeral container for each one of them with the Secret added as env variable using envFrom:

pod.Spec.EphemeralContainers = append(pod.Spec.EphemeralContainers, apiv1.EphemeralContainer{
	TargetContainerName: targetContainerName,
	EphemeralContainerCommon: apiv1.EphemeralContainerCommon{
		Image:   imageURL,
		Name:    ephemeralContainerName,
		Command: getCommand(),
		EnvFrom: []apiv1.EnvFromSource{
			{
				SecretRef: &apiv1.SecretEnvSource{
					LocalObjectReference: apiv1.LocalObjectReference{
						Name: name,
					},
					Optional: nil,
				},
			},
		},
		SecurityContext: &apiv1.SecurityContext{
			Privileged: &[]bool{true}[0],
		},
	},
})

clientset.CoreV1().Pods(metav1.NamespaceDefault).UpdateEphemeralContainers(
	ctx, pod.Name, &pod, metav1.UpdateOptions{},
)

When I run it, the ephemeral container is created but never started, it is left in Waiting state with Reason CreateContainerConfigError.

Ephemeral Containers:
  ephemeral1234567:
    Container ID:
    Image:         <image>
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command: <command>
    State:          Waiting
      Reason:       CreateContainerConfigError
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      secret-name  Secret  Optional: false
    Environment:     <none>
    Mounts:          <none>

The code seems to be correct since I can see the Secret is referenced in the Environment Variables.

Checking the events I can see the following:

Type	Reason	Age	From	Message
Warning	Failed	3s	kubelet	Error: failed to sync secret cache: timed out waiting for the condition

It seems the cache is not synced when an ephemeral container is created in a pod and therefore its creation fails because the Secret is not present in the cache.

Based on that previous assumption, I tried referencing the secret in the regular container defined in the deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mytarget
spec:
  selector:
    matchLabels:
      mylabel: mytarget
  replicas: 2
  template:
    metadata:
      labels:
        mylabel: mytarget
    spec:
      serviceAccountName: default
      containers:
        - name: mytarget
          image: <custom-image>
          command: ["sleep", "100d"]
          envFrom:
          - secretRef:
              name: secret-name

Running the same code that creates the ephemeral container is now run successfully and it creates the container with the secret as an environment variable.

exp123456:
    Container ID:  containerd://2398e96b9add961fd292d127d9e383b98c0c88100f87e0b8c7a8ff46aa1becdc
    Image:         <image>
    Image ID:      <imageId>
    Port:          <none>
    Host Port:     <none>
    Command: <command>
    State:          Running
      Started:      Wed, 11 Jan 2023 13:28:57 +0000
    Ready:          False
    Restart Count:  0
    Environment Variables from:
      manual-secret  Secret  Optional: false
    Environment:     <none>
    Mounts:          <none>

What did you expect to happen?

I would expect the ephemeral container to be created and successfully started with the appropriate environment variable containing the data in the secret.

How can we reproduce it (as minimally and precisely as possible)?

Using the code provided in the explanation of the issue should be enough to reproduce the problem.

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mytarget
spec:
  selector:
    matchLabels:
      mylabel: mytarget
  replicas: 2
  template:
    metadata:
      labels:
        mylabel: mytarget
    spec:
      serviceAccountName: default
      containers:
        - name: mytarget
          image: <custom-image>
          command: ["sleep", "100d"]

Secret:

apiVersion: v1
kind: Secret
metadata:
  name: manual-secret
type: Opaque
stringData:
  token: test1234

Request to update the Pod to create an ephemeral container:

PATCH http://localhost:8080/api/v1/namespaces/{namespace}/pods/{podname}/ephemeralcontainers

Add the following ephemeral container

ephemeralContainers:
  - name: ephemeral-uid-123456
    image: nginx
    command:
    - "sleep 100d"
    envFrom:
    - secretRef:
        name: manual-secret
    resources: {}
    terminationMessagePath: "/dev/termination-log"
    terminationMessagePolicy: File
    imagePullPolicy: IfNotPresent
    securityContext:
      privileged: true
    targetContainerName: mytarget

Anything else we need to know?

No response

Kubernetes version

Details

$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"24+", GitVersion:"v1.24.7-eks-fb459a0", GitCommit:"c240013134c03a740781ffa1436ba2688b50b494", GitTreeState:"clean", BuildDate:"2022-10-24T20:36:26Z", GoVersion:"go1.18.7", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.26) and server (1.24) exceeds the supported minor version skew of +/-1

Cloud provider

Details

OS version

Details

# On Linux:
$ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.17.1
PRETTY_NAME="Alpine Linux v3.17"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
$ uname -a
Linux pod-6bb864f684-tl5l7 5.4.219-126.411.amzn2.x86_64 #1 SMP Wed Nov 2 17:44:17 UTC 2022 x86_64 Linux

Install tools

Details

Container runtime (CRI) and version (if applicable)

Details

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

[pacoxu]

/sig node

[SergeyKanzhelev]

Just a wild guess, can this be related to this bug: #114167?

[SergeyKanzhelev]

@AdriaGS is it 100% repro or there should be special conditions to repro this issue?

/triage needs-information

[liggitt]

I think the issue is the node authorizer not considering that pod updates might add a new secret reference

kubernetes/plugin/pkg/auth/authorizer/node/graph_populator.go

Lines 80 to 92 in 97bbf07

	if oldPod, ok := oldObj.(*corev1.Pod); ok && oldPod != nil {
	if (pod.Spec.NodeName == oldPod.Spec.NodeName) && (pod.UID == oldPod.UID) {
	// Node and uid are unchanged, all object references in the pod spec are immutable
	klog.V(5).Infof("updatePod %s/%s, node unchanged", pod.Namespace, pod.Name)
	return
	}
	}
	klog.V(4).Infof("updatePod %s/%s for node %s", pod.Namespace, pod.Name, pod.Spec.NodeName)
	startTime := time.Now()
	g.graph.AddPod(pod)
	klog.V(5).Infof("updatePod %s/%s for node %s completed in %v", pod.Namespace, pod.Name, pod.Spec.NodeName, time.Since(startTime))
	}

A pod update that adds a new container with a new envFrom referencing a new secret needs to call g.graph.AddPod so the node gets access to that secret

Do you see authorization errors for that secret in the kubelet log?

/sig auth

[enj]

/assign

Sounds fun :)