pvc provisioning not working as expected

[lacisz491]

I want to use storageos on home kubernetes cluster but it did not works.

Detailed information:
KVM guests with Ubuntu 18.04

k8s-master:~# uname -a
Linux k8s-master 4.15.0-50-generic #54-Ubuntu SMP Mon May 6 18:46:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes version:

k8s-master:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

k8s-worker:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

Docker version:

k8s-master:~# docker version
Client:
 Version:           18.09.6
...
Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8

k8s-worker:~# docker version
Client:
 Version:           18.09.6
...
Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8

Cluster information

k8s-master:~# kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k8s-master   Ready    master   10d   v1.14.2   192.168.100.192   <none>        Ubuntu 18.04.2 LTS   4.15.0-50-generic   docker://18.9.6
k8s-worker   Ready    <none>   10d   v1.14.2   192.168.100.193   <none>        Ubuntu 18.04.2 LTS   4.15.0-50-generic   docker://18.9.6

I installed the storageos based on the documentation.
Steps:

git clone https://github.com/storageos/deploy.git storageos
cd storageos/k8s/deploy-storageos/cluster-operator
./deploy-operator.sh

then verified the pod:
kubectl -n storageos-operator get pod

I created a secret and a cluster definition:

kubectl create -f - <<END
apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "default"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  apiUsername: ********
  apiPassword: ********
END

then

kubectl create -f - <<END
apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
spec:
  secretRefName: "storageos-api" # Reference the Secret created in the previous step
  secretRefNamespace: "default"  # Namespace of the Secret
  images:
    nodeContainer: "storageos/node:1.2.1" # StorageOS version
  resources:
    requests:
    memory: "512Mi"
END

After these steps
i verified the storageclass and everything.

k8s-master:~# kubectl get all --all-namespaces |grep storage

storageos-operator   pod/storageoscluster-operator-77d8656c5d-6b94z   1/1     Running   1          3h22m
storageos            pod/storageos-daemonset-2dvl2                    1/1     Running   2          3h18m
storageos            pod/storageos-daemonset-4x82s                    1/1     Running   1          3h18m

storageos       service/storageos              ClusterIP   10.101.224.93    <none>        5705/TCP                     3h18m

storageos     daemonset.apps/storageos-daemonset       2         2         2       2            2           <none>                            3h18m

storageos-operator   deployment.apps/storageoscluster-operator   1/1     1            1           3h22m
storageos-operator   replicaset.apps/storageoscluster-operator-77d8656c5d   1         1         1       3h22m

Finally:

kubectl apply -f k8s/examples/pvc.yaml -n smoke
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-1
  annotations:
    volume.beta.kubernetes.io/storage-class: fast
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

k8s-master:~/storageos# kubectl get pvc -n smoke
NAME    STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1   Pending                                      fast           160m

k8s-master:~/storageos# kubectl describe  pvc pvc-1  -n smoke
Name:          pvc-1
Namespace:     smoke
StorageClass:  fast
Status:        Pending
Volume:
Labels:        <none>
Annotations:   kubectl.kubernetes.io/last-applied-configuration:
                 {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volume.beta.kubernetes.io/storage-class":"fast"},"name":"pvc...
               volume.beta.kubernetes.io/storage-class: fast
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/storageos
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Events:
  Type       Reason              Age                  From                         Message
  ----       ------              ----                 ----                         -------
  Warning    ProvisioningFailed  25m (x34 over 161m)  persistentvolume-controller  Failed to provision volume with StorageClass "fast": API error (Requested object not found. Does this item exist?): 404 page not found
  Warning    ProvisioningFailed  46s (x44 over 161m)  persistentvolume-controller  Failed to provision volume with StorageClass "fast": API error (Unauthenticated access of secure endpoint, please retry after authentication): Failed to find account
Mounted By:  <none>

I think i do everything by default.. so i really dont understand why can not auth when every secret is on good place and i can use the storageos-cli.

export STORAGEOS_USERNAME=storageadmin
export STORAGEOS_PASSWORD=*****
export STORAGEOS_HOST=10.101.224.93

k8s-master:~/storageos# storageos cluster health
NODE        CP_STATUS  DP_STATUS
k8s-master  Healthy    Healthy

but

storageos volume ls
API error (Unauthenticated access of secure endpoint, please retry after authentication): no token present in request

Please help me to move forward this issue.

[lacisz491]

update:
It looks like, after the cleanup and new install it works. but i did some changes.
When i created the secret and the StorageOSCluster, i did these on not in project default.. i did in project storageos-operator.

[avestuk]

Hi @lacisz491, glad to hear you've got StorageOS working. You specify the secretRefName and secretRefNamespace in the StorageOSCluster CR so you can create them in a different namespace they just need to match.