New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OCP 3.10 - Local volume not allocated to a claim. #21108

Open
whitingjr opened this Issue Sep 26, 2018 · 4 comments

Comments

Projects
None yet
5 participants
@whitingjr

whitingjr commented Sep 26, 2018

[provide a description of the issue]

The local volumes created by the provisioner are not being allocated to a claim.

Version

oc version

oc v3.10.34
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://benchserver7:8443
openshift v3.10.34
kubernetes v1.10.0+b81c8f8

Steps To Reproduce
  1. install ocp 3.10 from the branch in Git
  2. configure the local volume provisioner
  3. configure at least 1 pv
  4. create a pod with a pvc that uses the storage class configured in step 2.
Current Result

Claims stuck in Pending state.

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
backend-redis-storage Pending local-ssd 39m
mysql-storage Pending local-ssd 1h
system-redis-storage Pending local-ssd 1h
system-storage Bound pvnfs.system-storage 100Mi RWX 6d

Expected Result

All claims allocated a persistent volume.

Additional Information

docker logs 9fd26a1352e5

I0926 12:55:56.295183 1 common.go:255] StorageClass "local-ssd" configured with MountDir "/mnt/local-storage/local-ssd", HostDir "/mnt/local-storage/local-ssd", BlockCleanerCommand ["/scripts/quick_reset.sh"]
I0926 12:55:56.295244 1 main.go:42] Configuration parsing has been completed, ready to run...
I0926 12:55:56.295407 1 common.go:315] Creating client using in-cluster config
I0926 12:55:56.309038 1 main.go:52] Starting controller
I0926 12:55:56.309056 1 controller.go:42] Initializing volume cache
I0926 12:55:56.311407 1 populator.go:85] Starting Informer controller
I0926 12:55:56.311425 1 populator.go:89] Waiting for Informer initial sync
I0926 12:55:56.322058 1 cache.go:55] Added pv "local-pv-3d1345ab" to cache
I0926 12:55:56.322076 1 cache.go:55] Added pv "local-pv-850e763b" to cache
I0926 12:55:56.322082 1 cache.go:55] Added pv "local-pv-ca9ccf2" to cache
I0926 12:55:57.311643 1 controller.go:72] Controller started
W0926 13:13:39.418510 1 reflector.go:341] github.com/kubernetes-incubator/external-storage/local-volume/provisioner/pkg/populator/populator.go:87: watch of *v1.PersistentVolume ended with: The resourceVersion for the provided watch is too old.
I0926 13:13:40.421862 1 cache.go:64] Updated pv "local-pv-3d1345ab" to cache
I0926 13:13:40.421894 1 cache.go:64] Updated pv "local-pv-850e763b" to cache
I0926 13:13:40.421904 1 cache.go:64] Updated pv "local-pv-ca9ccf2" to cache

@whitingjr

This comment has been minimized.

Show comment
Hide comment
@whitingjr

whitingjr Oct 2, 2018

Found the cause. Which I suspect is a defect.
The PVC was requesting 1014Mi storage size and the PV was 1014Mi in size. That's the mounted size of the mountpoint.

# df -kh
/dev/mapper/vglocalssdpool-lvsystem--redis--storage  1014M   33M  982M   4% /mnt/local-storage/local-ssd/system-redis-storage

What I found was the PVC and PV were never successfully associated by Openshift. The status of the PVC/PV was never updated to Bound.

apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: null
  finalizers:
  - kubernetes.io/pvc-protection
  name: system-redis-storage
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: 1014Mi
  storageClassName: local-ssd

The workaround for me was to re-create the same PVC with a smaller size. Adjusting it from 1014Mi to 500Mi size.

apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: null
  finalizers:
  - kubernetes.io/pvc-protection
  name: system-redis-storage
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: 500Mi
  storageClassName: local-ssd

After doing that the PVC and PV were associated and the status of both set to Bound.

whitingjr commented Oct 2, 2018

Found the cause. Which I suspect is a defect.
The PVC was requesting 1014Mi storage size and the PV was 1014Mi in size. That's the mounted size of the mountpoint.

# df -kh
/dev/mapper/vglocalssdpool-lvsystem--redis--storage  1014M   33M  982M   4% /mnt/local-storage/local-ssd/system-redis-storage

What I found was the PVC and PV were never successfully associated by Openshift. The status of the PVC/PV was never updated to Bound.

apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: null
  finalizers:
  - kubernetes.io/pvc-protection
  name: system-redis-storage
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: 1014Mi
  storageClassName: local-ssd

The workaround for me was to re-create the same PVC with a smaller size. Adjusting it from 1014Mi to 500Mi size.

apiVersion: "v1"
kind: "PersistentVolumeClaim"
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  creationTimestamp: null
  finalizers:
  - kubernetes.io/pvc-protection
  name: system-redis-storage
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: 500Mi
  storageClassName: local-ssd

After doing that the PVC and PV were associated and the status of both set to Bound.

@jwforres

This comment has been minimized.

Show comment
Hide comment
@jwforres
Member

jwforres commented Oct 11, 2018

@jsafrane

This comment has been minimized.

Show comment
Hide comment
@jsafrane

jsafrane Oct 12, 2018

Contributor

@whitingjr, it seems the provisioner created some PV, so at least that part is somewhat working. Can you please share yaml of the created PV? It seems its capacity is wrong. How wrong is it?

Contributor

jsafrane commented Oct 12, 2018

@whitingjr, it seems the provisioner created some PV, so at least that part is somewhat working. Can you please share yaml of the created PV? It seems its capacity is wrong. How wrong is it?

@whitingjr

This comment has been minimized.

Show comment
Hide comment
@whitingjr

whitingjr Oct 15, 2018

@jsafrane here is the pv that is Bound and it has the size 1014Mi. Identical to the initial claim size that failed.

# oc get pv|grep backend-redis-storage
local-pv-f4098e94      1014Mi     RWO            Delete           Bound       3scale/backend-redis-storage   local-ssd                13d
# oc get pv local-pv-f4098e94 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: local-volume-provisioner-benchserver2-1b59f74b-ad19-11e8-ac02-941882782994
  creationTimestamp: 2018-10-01T15:28:39Z
  finalizers:
  - kubernetes.io/pv-protection
  name: local-pv-f4098e94
  resourceVersion: "4350875"
  selfLink: /api/v1/persistentvolumes/local-pv-f4098e94
  uid: abee5137-c58e-11e8-895d-941882782994
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1014Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: backend-redis-storage
    namespace: 3scale
    resourceVersion: "4349759"
    uid: 8eb369b4-c594-11e8-895d-941882782994
  local:
    path: /mnt/local-storage/local-ssd/mysql-storage
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - benchserver2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-ssd
status:
  phase: Bound

What's interesting is the pv claim. Dumping out that claim you see the storage size as 1014Mi not the 500Mi specified in the workaround.

# oc get pvc backend-redis-storage -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: 2018-10-01T16:10:47Z
  finalizers:
  - kubernetes.io/pvc-protection
  name: backend-redis-storage
  namespace: 3scale
  resourceVersion: "4350878"
  selfLink: /api/v1/namespaces/3scale/persistentvolumeclaims/backend-redis-storage
  uid: 8eb369b4-c594-11e8-895d-941882782994
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: local-ssd
  volumeName: local-pv-f4098e94
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1014Mi
  phase: Bound
#

whitingjr commented Oct 15, 2018

@jsafrane here is the pv that is Bound and it has the size 1014Mi. Identical to the initial claim size that failed.

# oc get pv|grep backend-redis-storage
local-pv-f4098e94      1014Mi     RWO            Delete           Bound       3scale/backend-redis-storage   local-ssd                13d
# oc get pv local-pv-f4098e94 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/bound-by-controller: "yes"
    pv.kubernetes.io/provisioned-by: local-volume-provisioner-benchserver2-1b59f74b-ad19-11e8-ac02-941882782994
  creationTimestamp: 2018-10-01T15:28:39Z
  finalizers:
  - kubernetes.io/pv-protection
  name: local-pv-f4098e94
  resourceVersion: "4350875"
  selfLink: /api/v1/persistentvolumes/local-pv-f4098e94
  uid: abee5137-c58e-11e8-895d-941882782994
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1014Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: backend-redis-storage
    namespace: 3scale
    resourceVersion: "4349759"
    uid: 8eb369b4-c594-11e8-895d-941882782994
  local:
    path: /mnt/local-storage/local-ssd/mysql-storage
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - benchserver2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-ssd
status:
  phase: Bound

What's interesting is the pv claim. Dumping out that claim you see the storage size as 1014Mi not the 500Mi specified in the workaround.

# oc get pvc backend-redis-storage -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
  creationTimestamp: 2018-10-01T16:10:47Z
  finalizers:
  - kubernetes.io/pvc-protection
  name: backend-redis-storage
  namespace: 3scale
  resourceVersion: "4350878"
  selfLink: /api/v1/namespaces/3scale/persistentvolumeclaims/backend-redis-storage
  uid: 8eb369b4-c594-11e8-895d-941882782994
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 500Mi
  storageClassName: local-ssd
  volumeName: local-pv-f4098e94
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1014Mi
  phase: Bound
#
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment