Can't follow tutorial to get a blob container to mount

[bbearce]

What happened:
Followed tutorial at this exact link: https://github.com/kubernetes-sigs/blob-csi-driver/blob/master/deploy/example/e2e_usage.md

What you expected to happen:
Under section "Static Provisioning(use an existing storage account)" there are two options:

[1] Using storage class
[2] Use secret

I've tried both and the nginx demo deploy never creates let alone mounts the azure blob container. Below are my steps for section [2]. This is the indefinite state my nginx example pod is in that is supposed to mount an azure blob container.

$ kubectl get pods
NAME                                   READY   STATUS              RESTARTS   AGE
ingress-azure-7658ffc594-skg8w         1/1     Running             0          43d
mic-6b5468dc47-xrljr                   1/1     Running             0          43d
mic-6b5468dc47-z98nd                   1/1     Running             0          43d
nginx-blob                             0/1     ContainerCreating   0          2m50s 

This shows the current list of pods already running. Below is the output from: $ kubectl describe pods nginx-blob

Events:
  Type     Reason              Age    From                     Message
  ----     ------              ----   ----                     -------
  Normal   Scheduled           3m49s  default-scheduler        Successfully assigned default/nginx-blob to aks-nodepool1-30745680-vmss000004
  Warning  FailedAttachVolume  109s   attachdetach-controller  AttachVolume.Attach failed for volume "pv-blob" : attachdetachment timeout for volume uniqe-volumeid
  Warning  FailedMount         106s   kubelet                  Unable to attach or mount volumes: unmounted volumes=[blob01], unattached volumes=[blob01 default-token-br7l4]: timed out waiting for the condition

How to reproduce it:

azurestorageaccountname="mghradcodalabdest"
azurestorageaccountkey="CP5bLexed6eglvOqH4Jmdm*******..."

kubectl create secret generic azure-secret --from-literal azurestorageaccountname=$azurestorageaccountname --from-literal azurestorageaccountkey=$azurestorageaccountkey --type=Opaque

# First of two attempts
wget https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/pvc-blob-csi-static.yaml
kubectl delete -f pvc-blob-csi-static.yaml
kubectl create -f pvc-blob-csi-static.yaml

wget https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/pv-blobfuse-csi.yaml
kubectl delete -f pv-blobfuse-csi.yaml
kubectl create -f pv-blobfuse-csi.yaml

wget https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/nginx-pod-blob.yaml
kubectl delete -f nginx-pod-blob.yaml
kubectl create -f nginx-pod-blob.yaml

In that attempt I used the downloaded pv-blobfuse-csi.yaml file (linked by "Create PV: download pv-blobfuse-csi.yaml file") and changed these values under volumeAttributes:
resourceGroup: mgh-rad-codalab-d-e-storage-rg
storageAccount: mghradcodalabdest
containerName: var

Next I will use the text displayed on the tutorial:

# Second of two attempts
wget https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/pvc-blob-csi-static.yaml
kubectl delete -f pvc-blob-csi-static.yaml
kubectl create -f pvc-blob-csi-static.yaml

#You'll notice in the instructions that linked from text "Create PV: download pv-blobfuse-csi.yaml file" is a file that is actually not the same as the referenced PV example you can copy and paste. Using the text displayed for file pv-blobfuse-csi.yaml.
kubectl delete -f pv-blobfuse-csi.yaml
kubectl create -f pv-blobfuse-csi.yaml

wget https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/deploy/example/nginx-pod-blob.yaml
kubectl delete -f nginx-pod-blob.yaml
kubectl create -f nginx-pod-blob.yaml

but still I get the same error.

Events:
  Type     Reason              Age                    From                     Message
  ----     ------              ----                   ----                     -------
  Warning  FailedMount         7m50s (x43 over 127m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[blob01], unattached volumes=[blob01 default-token-br7l4]: timed out waiting for the condition
  Warning  FailedMount         61s (x12 over 121m)    kubelet                  Unable to attach or mount volumes: unmounted volumes=[blob01], unattached volumes=[default-token-br7l4 blob01]: timed out waiting for the condition
  Warning  FailedAttachVolume  51s (x36 over 127m)    attachdetach-controller  AttachVolume.Attach failed for volume "pv-blob" : attachdetachment timeout for volume uniqe-volumeid

Anything else we need to know?:

Environment:

• CSI Driver version: I'm not sure how to find it. There is no quick command for csi driers like everything else. For the csi pod I get this from running kubectl describe pod csi-secrets-store-csi -driver-n698q --namespace csi:

Labels:       app=secrets-store-csi-driver
              app.kubernetes.io/instance=csi
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=secrets-store-csi-driver
              app.kubernetes.io/version=0.0.20
              controller-revision-hash=767dc45cd
              helm.sh/chart=secrets-store-csi-driver-0.0.20
              pod-template-generation=1

My Helm install commands:

helm repo add blob-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/blob-csi-driver/master/charts
helm install blob-csi-driver blob-csi-driver/blob-csi-driver --namespace kube-system

• Kubernetes version (use kubectl version): v1.19.2
• OS (e.g. from /etc/os-release):

NAME="Pop!_OS"
VERSION="20.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 20.04 LTS"
VERSION_ID="20.04"

• Kernel (e.g. uname -a):

Linux pop-os 5.11.0-7614-generic #15~1618626693~20.04~ecb25cd-Ubuntu SMP Thu Apr 22 16:20:32 UTC  x86_64 x86_64 x86_64 GNU/Linux
```1
- Install tools: ?
- Others: The master branch has a little x next to it making it seem like some checks haven't passed. Should be concerned about that? Should I be experimenting with other branches other than master?

[andyzhangx]

I think the error is because you already have a pv with volumeHandle: uniqe-volumeid, make sure this volumeid is unique in the cluster

[bbearce]

I have this for pv\pvc:

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE
pv-blob   10Gi       RWX            Retain           Bound    default/pvc-blob                           3h42m

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kubectl get pvc
NAME       STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-blob   Bound    pv-blob   10Gi       RWX                           3h42m

So only these two pairs and I believe they are freshly made within the last two hours. I deleted them and recreated with new volumeHandle: uniqe-volumeid-new.

My new pv\pvc statuses:

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kubectl get pv
NAME      CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM              STORAGECLASS   REASON   AGE
pv-blob   10Gi       RWX            Retain           Bound    default/pvc-blob                           2m24s
bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kubectl get pvc
NAME       STATUS   VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-blob   Bound    pv-blob   10Gi       RWX                           3m34s

Could there be pv\pvcs I can't see? I read that unlike namespaced objects that pv\pvc\sc objects are available to the whole cluster, so when I run kubectl get pv or kubectl get pv and only see one record each, I feel like that has to be all of them.

May I check an assumption? Storage classes are used with cluster identities correct? That should mean I need either a sc\pvc pair or a pv\pvc pair right? All three are not necessary at the same time as either the storage class in conjunction with identities or the pv with cluster secrets provide the necessary credentials for securely accessing the azure blob. Is this correct?

In the end my new error for describing the nginx pod is as follows:

Events:
  Type     Reason              Age                   From                     Message
  ----     ------              ----                  ----                     -------
  Normal   Scheduled           6m24s                 default-scheduler        Successfully assigned default/nginx-blob to aks-nodepool1-30745680-vmss000004
  Warning  FailedMount         2m6s (x2 over 4m20s)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[blob01], unattached volumes=[blob01 default-token-br7l4]: timed out waiting for the condition
  Warning  FailedAttachVolume  22s (x3 over 4m24s)   attachdetach-controller  AttachVolume.Attach failed for volume "pv-blob" : attachdetachment timeout for volume uniqe-volumeid-new

[andyzhangx]

the error attachdetach-controller AttachVolume.Attach failed for volume "pv-blob" : attachdetachment timeout for volume uniqe-volumeid-new is returned by kube-controller-manager, is that component working?

I think you could try dynamic provisioning for narrow down, I doubt that won't work either.

[bbearce]

Silly question perhaps, but when I run kube-controller-manager I get this:

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kube-controller-manager
 
Command 'kube-controller-manager' not found, but can be installed with:
 
sudo snap install kube-controller-manager

Should this come with the cluster by default? It sounds like something that must already be running and would be an integral part of the system. I can easily install it and am about to, but it strikes me as odd that I wouldn't have it already. I am installling now, but want to check assumptions as a control loop to monitor the system seems pretty key to a state-based system like kubernetes.

[andyzhangx]

kube-controller-manager should be installed with cluster, run kubectl get po --all-namespaces | grep controller-manager to check

[bbearce]

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kubectl get po --all-namespaces | grep controller-manager

I am concerned this returned nothing as when I installed I got this:

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ sudo snap install kube-controller-manager
[sudo] password for bbearce:
kube-controller-manager 1.21.0 from Canonical✓ installed

Thoughts? By the way, thanks a ton for helping! This means a lot. Is there a chance what I installed is some local client version for monitoring?

BTW without "grep" I get this for that command:

bbearce@pop-os:~/Desktop/deploy_with_Derek/competition-infrastructure/PVC_Practice$ kubectl get po --all-namespaces           
NAMESPACE      NAME                                         READY   STATUS              RESTARTS   AGE
cert-manager   cert-manager-57847c454c-2q75h                1/1     Running             0          43d
cert-manager   cert-manager-cainjector-5594d9d574-4w6rs     1/1     Running             0          43d
cert-manager   cert-manager-webhook-5f49df95f4-lxgpj        1/1     Running             0          43d
csi            csi-csi-secrets-store-provider-azure-mp2dl   1/1     Running             0          35h
csi            csi-secrets-store-csi-driver-n698q           3/3     Running             0          35h
default        ingress-azure-7658ffc594-skg8w               1/1     Running             0          43d
default        mic-6b5468dc47-xrljr                         1/1     Running             0          43d
default        mic-6b5468dc47-z98nd                         1/1     Running             0          43d
default        nginx-blob                                   0/1     ContainerCreating   0          28m
default        nmi-5qswx                                    1/1     Running             0          35h
default        nvidia-device-plugin-daemonset-nxspm         1/1     Running             0          35h
kube-system    azure-cni-networkmonitor-h2jvf               1/1     Running             0          35h
kube-system    azure-ip-masq-agent-64k64                    1/1     Running             0          35h
kube-system    blobfuse-flexvol-installer-dqds8             1/1     Running             0          35h
kube-system    coredns-autoscaler-599949fd86-fpj7h          1/1     Running             0          35h
kube-system    coredns-b94d8b788-86rm4                      1/1     Running             0          43d
kube-system    coredns-b94d8b788-9twcv                      1/1     Running             0          43d
kube-system    kube-proxy-rkqx8                             1/1     Running             0          35h
kube-system    metrics-server-77c8679d7d-v7kxf              1/1     Running             0          43d
kube-system    omsagent-gvlfp                               1/1     Running             1          35h
kube-system    omsagent-rs-94976f9f6-cf4zh                  1/1     Running             0          35h
kube-system    tunnelfront-5b9664776d-m98lg                 1/1     Running             0          35h

[andyzhangx]

issue resolved?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.