Jobs which mounted to the same Azure-Blob can not run on the same node

[Binyang2014]

When two jobs which mounted to the same azure-blob scheduled to the same node, one job will be stuck in waiting status.
And the error log is: Error: temp directory '/tmp/blobfuse' is not empty. blobfuse needs an empty temp directory

This is a bug of blob flex-volume: https://github.com/Azure/kubernetes-volume-drivers/blob/3f77fd816170083b2575af5858eb3b47368c7f65/flexvolume/blobfuse/deployment/blobfuse-flexvol-installer/blobfuse#L107 , all jobs using the same temp path. So only the first comes job will mounted successfully.

Consider to use blobfuse-csi-driver instead of blob-flex-volume.
https://github.com/kubernetes-sigs/blobfuse-csi-driver

[abuccts]

As a workaround, add the entrypoint (command and args) in flexvolume driver, it'll insert a patch in binary.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: blobfuse-flexvol-installer
  namespace: kube-system
  labels:
    k8s-app: blobfuse
spec:
  selector:
    matchLabels:
      name: blobfuse
  template:
    metadata:
      labels:
        name: blobfuse
    spec:
      containers:
        - name: blobfuse-flexvol-installer
          image: mcr.microsoft.com/k8s/flexvolume/blobfuse-flexvolume:1.0.13
          imagePullPolicy: IfNotPresent
          command: ["/bin/sh"]
          args: ["-c", "sed -i '101i TMP_PATH=$(echo ${TMP_PATH}-$(date +%N))' /blobfuse/blobfuse && /bin/install_blobfuse_flexvol.sh"],
          volumeMounts:
            - name: volplugins
              mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
            - name: varlog
              mountPath: /var/log/
      volumes:
        - name: varlog
          hostPath:
            path: /var/log/
        - name: volplugins
          hostPath:
            path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
      nodeSelector:
        beta.kubernetes.io/os: linux

[fanyangCS]

this is a bug in azureblob. When installing blob driver, customers are recommended to use the above yaml to deploy the flexvolume driver to their k8s cluster.

[Binyang2014]

According to Azure/kubernetes-volume-drivers#66 (comment) Azure blob flexvolume is in maintenance mode, and Azure blob CSI is recommended.

[fanyangCS]

#4644

[andyzhangx]

offer you another workaround, downgrade blobfuse binary to 1.1.1 :
Azure/kubernetes-volume-drivers#66 (comment)