The sidecar container is at the spec.containers[0] position which may cause issues in some workloads

[songjiaxun]

The current sidecar container injection logic will inject the sidecar container at the spec.containers[0] position in a Pod.

The design makes sure that kubelet or the container runtime will start up the sidecar container first to obtain the file descriptor of the mount point. Otherwise, the other containers that consume the mount point will hang in the start up step.

This design may break some application's assumption, for example, the Airflow Kubernetes Executor assumes that the base container is always at the spec.containers[0] position in the worker Pod, which conflicts with the sidecar container injection logic.

We are waiting for the KEP for sidecar container pattern: kubernetes/enhancements#3761. After the KEP is accepted and implemented, we will migrate to leverage the native supported sidecar container pattern.

A workaround is to modify the workload Pod to manually inject the sidecar container. Then the sidecar container can be at any position in the container array. However, if the sidecar container is at a later position than the workload container that consumes the volume, the workload container start up will hang 2 minutes and timeout. But the sidecar container start up will proceed. Whenever the sidecar container starts up, the workload container can start as usual.

[songjiaxun]

Here is a workaround with limitations:

In order to simulate the sidecar injection achieved by the webhook, you can manually inject the gcsfuse sidecar container at any position in the container array. Meanwhile, you will also need to manually inject an emptyDir. Specifically, you need to specify the following components in your workload:

• An emptyDir volume named gke-gcsfuse-tmp.
• A container named gke-gcsfuse-sidecar, using the sidecar mounter image gke.gcr.io/gcs-fuse-csi-driver-sidecar-mounter:v0.1.3-gke.0@sha256:854e1aa1178dc3f7e3ec5fa03cea5e32f0385ff6230efd836a22e86beb876740, and consuming the emptyDir volume gke-gcsfuse-tmp

Here is an example:

apiVersion: v1
kind: Pod
metadata:
  name: sidecar-test
# Because you are manually injecting the sidecar container,
# you don't need the Pod annotation `gke-gcsfuse/volumes: "true"`.
# Actually, in order to suppress the webhook sidecar injection,
# you must NOT put the Pod annotation.
spec:
  serviceAccountName: gcs-csi
  containers:
  - name: busybox
    image: busybox
    resources:
      limits:
        cpu: 250m
        ephemeral-storage: 1Gi
        memory: 256Mi
      requests:
        cpu: 250m
        ephemeral-storage: 1Gi
        memory: 256Mi
    command:
      - "/bin/sh"
      - "-c"
      - sleep infinite
    volumeMounts:
    - name: gcs-fuse-csi-ephemeral
      mountPath: /data
# Instead of specifying the sidecar container at the first position in the container array,
# you can inject it at any position.
  - name: gke-gcsfuse-sidecar
    image: gke.gcr.io/gcs-fuse-csi-driver-sidecar-mounter:v0.1.4-gke.1@sha256:442969f1e565ba63ff22837ce7a530b6cbdb26330140b7f9e1dc23f53f1df335
    imagePullPolicy: IfNotPresent
    args:
    - --v=5
    resources:
      limits:
        cpu: 250m
        ephemeral-storage: 1Gi
        memory: 256Mi
      requests:
        cpu: 250m
        ephemeral-storage: 1Gi
        memory: 256Mi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      readOnlyRootFilesystem: true
      runAsGroup: 65534
      runAsNonRoot: true
      runAsUser: 65534
      seccompProfile:
        type: RuntimeDefault
    volumeMounts:
    - mountPath: /gcsfuse-tmp
      name: gke-gcsfuse-tmp
  volumes:
  - name: gcs-fuse-csi-ephemeral
    csi:
      driver: gcsfuse.csi.storage.gke.io
      volumeAttributes:
        bucketName: <your-bucket-name> # unique bucket name
  - emptyDir: {}
    name: gke-gcsfuse-tmp

---

Limitations

• This workaround is not thoroughly tested. Unless you have a urge need, please specify the Pod annotation and use the managed webhook to inject the sidecar container.
• Using this workaround, you will see your Pod stuck in a Pending status for about 2 minutes. Specifically, you will see a kubelet warning event with a message Error: context deadline exceeded. This is expected and inevitable. After 2 minutes, the kubelet should continue to schedule the Pod.
• This workaround is NOT officially supported by Google. Please test before you adopt it in your production environment.

[songjiaxun]

The KEP for sidecar container pattern https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers is implemented and merged: kubernetes/kubernetes#116429

The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination.

This change can be a good solution.

[daikeshi]

Hey @songjiaxun, since kubernetes/kubernetes#116429 is merged, is there any plan to address this issue?

[songjiaxun]

Hi @daikeshi , yes, we are targeting early April 2024 to adopt the native sidecar container feature starting with GKE 1.29 clusters.

[songjiaxun]

The new GKE version rollout completed. Starting from GKE 1.29.3-gke.1093000, the CSI driver injects the GCSFuse sidecar container as an init container.

To try out the new feature, please upgrade your GKE cluster to 1.29.3-gke.1093000 or later, and make sure ALL your nodes are also upgraded to GKE version 1.29 or later, then re-deploy your workloads.

Closing this issue for now.