This section describes how the GCS FUSE sidecar container is injected and how a GCS bucket-backed volume is mounted. It helps you understand the restrictions of this sidecar container mode design.
All the Pod creation requests are monitored by a webhook controller. If the Pod annotation gke-gcsfuse/volumes: "true" is detected, the webhook will inject the sidecar container at position 0 of the regular container array by modifying the Pod spec. The Cloud Storage FUSE processes run in the sidecar container.
After the Pod is scheduled onto a node, the GCS FUSE CSI Driver node server, which runs as a privileged container on each node, opens the /dev/fuse device on the node and obtains the file descriptor. Then the CSI driver calls mount.fuse3(8) passing the file descriptor via the mount option “fd=N” to create a mount point. In the end, the CSI driver calls sendmsg(2) to send the file descriptor to the sidecar container via Unix Domain Socket (UDS) SCM_RIGHTS.
After the CSI driver creates the mount point, it will inform kubelet to proceed with the Pod startup. The containers on the Pod spec will be started up in order, so the sidecar container will be started first.
In the sidecar container, which is an unprivileged container, a process connects to the UDS and calls recvmsg(2) to receive the file descriptor. Then the process calls Cloud Storage FUSE passing the file descriptor to start to serve the FUSE mount point. Instead of passing the actual mount point path, we pass the file descriptor to Cloud Storage FUSE as it supports the magic /dev/fd/N syntax. Before the Cloud Storage FUSE takes over the file descriptor, any operations against the mount point will hang.
Until the Cloud Storage FUSE takes over the file descriptor, the mount point is not accessible. Any operations against the mount point will hang, including stat(2) that is used to check if the mount point exists.
The sidecar container, or more precisely, the Cloud Storage FUSE process that serves the mount point needs to remain running for the full duration of the Pod's lifecycle. If the Cloud Storage FUSE process is killed, the workload application will throw IO error Transport endpoint is not connected.
- The CSI driver does not support volumes for initContainers
- The sidecar container is at the spec.containers[0] position which may cause issues in some workloads
- subPath does not work when Anthos Service Mesh is enabled
- "Error: context deadline exceeded" when Anthos Service Mesh is enabled
- The sidecar container does not work well with istio-proxy sidecar container
Unfortunately, there is no good short-term solution or workaround for the above issues due to the restrictions of the sidecar container mode design.
The sidecar containers KEP is implemented in this PR Add SidecarContainers feature.
The new feature gate "SidecarContainers" is now available. This feature introduces sidecar containers, a new type of init container that starts before other containers but remains running for the full duration of the pod's lifecycle and will not block pod termination.
This new feature is a good long-term solution. Instead of injecting the sidecar container as a regular container, we will leverage the new SidecarContainers feature to inject the container as an init container, so that other non-sidecar init container can also use the CSI driver.
We are currently testing the SidecarContainers feature, and will adopt the feature when it is available on GKE.