[pvresize]Display of pvc capacity did not make corresponding changes when pv resized #61259

taoyu27 · 2018-03-16T03:42:46Z

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

What happened:
Display of pvc capacity did not make corresponding changes when pv resized

What you expected to happen:
display of pvc capacity did make corresponding changes when pv resized

How to reproduce it (as minimally and precisely as possible):

create pvc, request storage: 1Gi

apiVersion: v1
metadata:
  name: helloworld-claim-demo
  annotations:
    volume.beta.kubernetes.io/storage-class: "rbd-demo"
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd-demo
  resources:
    requests:
      storage: 1Gi

kubectl get pvc
NAME                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
helloworld-claim-demo   Bound     pvc-0da06c71-2832-11e8-8857-fa163e4e23c0   1Gi        RWO            rbd-demo       18h

kubectl edit pvc, make request storage 4Gi

update pvc.yaml file
apiVersion: v1
metadata:
  name: helloworld-claim-demo
  annotations:
    volume.beta.kubernetes.io/storage-class: "rbd-demo"
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rbd-demo
  resources:
    requests:
      storage: 4Gi


kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM                           STORAGECLASS   REASON    AGE
pvc-0da06c71-2832-11e8-8857-fa163e4e23c0   4Gi        RWO            Delete           Bound     default/helloworld-claim-demo   rbd-demo                 18h

check the display of pvc capacity

kubectl get pvc
NAME                    STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
helloworld-claim-demo   Bound     pvc-0da06c71-2832-11e8-8857-fa163e4e23c0   1Gi       RWO            rbd-demo       18h

the display of pvc capacity is still 1Gi...

Environment:

Kubernetes version (use kubectl version):
1.10 bata 4
OS (e.g. from /etc/os-release):
centos 7.2
Kernel (e.g. uname -a):
3.10.0-327.el7.x86_64 Unit test coverage in Kubelet is lousy. (~30%) #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Others:

The text was updated successfully, but these errors were encountered:

taoyu27 · 2018-03-16T08:43:58Z

/sig storage

gnufied · 2018-03-16T10:30:55Z

/assign

leon-g-xu · 2018-04-26T01:26:00Z

I am also hitting this issue. Any update? On the node itself I can see that the mounted volume capacity has been updated though. Is the pvc capacity only a label or does it reflect the underlying available storage capacity mounted in the container?

leon-g-xu · 2018-04-26T01:36:00Z

Also is there any manual workaround for this issue? We are on k8s 1.9.4

etienn01 · 2018-05-04T07:40:32Z

Seeing the same issue on k8s 1.9.3.
I enabled ExpandPersistentVolumes feature-gate, edited the StorageClass to set allowVolumeExpansion and then increased the spec.resources.requests.storage in a PVC. The PV size has been increased (also the corresponding AWS EBS volume has the right size now) but the change is not reflected on the PVC.

etienn01 · 2018-05-07T16:49:46Z

Found a way to get it to work thanks to this comment:

run resize2fs after EBS volume resize has completed to extend your partition (you can use lsblk to check the actual volume size)
use etcdctl to update the claim status with the new size

gnufied · 2018-06-12T20:40:10Z

This could be because for something like RBD, for entire resizing operation to complete, you must [re]create a pod that was using the PVC. Likely the resize operation is just waiting for file system resize to be completed on the node. In 1.11, the PVC will have condition FileSystemResizePending when this happens.

Also in 1.11 we will be moving this feature to beta. @guodongx @etienn01 can you guys retest this with 1.11 and report?

wminshew · 2018-08-24T16:57:30Z

Doing this manually with k8s 1.9, but having trouble with resize2fs. Some outputs below:

/ # lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda       8:0    0  100G  0 disk 
├─sda1    8:1    0 95.9G  0 part /etc/hosts
├─sda2    8:2    0   16M  0 part 
├─sda3    8:3    0    2G  0 part 
├─sda4    8:4    0   16M  0 part 
├─sda5    8:5    0    2G  0 part 
├─sda6    8:6    0  512B  0 part 
├─sda7    8:7    0  512B  0 part 
├─sda8    8:8    0   16M  0 part 
├─sda9    8:9    0  512B  0 part 
├─sda10   8:10   0  512B  0 part 
├─sda11   8:11   0    8M  0 part 
└─sda12   8:12   0   32M  0 part 
sdb       8:16   0   10G  0 disk /devpi
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                  94.3G      2.5G     91.8G   3% /
tmpfs                     1.8G         0      1.8G   0% /dev
tmpfs                     1.8G         0      1.8G   0% /sys/fs/cgroup
/dev/sdb                975.9M    619.0M    289.7M  68% /devpi
/dev/sda1                94.3G      2.5G     91.8G   3% /dev/termination-log
/dev/sda1                94.3G      2.5G     91.8G   3% /etc/resolv.conf
/dev/sda1                94.3G      2.5G     91.8G   3% /etc/hostname
/dev/sda1                94.3G      2.5G     91.8G   3% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                     1.8G     12.0K      1.8G   0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs                     1.8G         0      1.8G   0% /proc/kcore
tmpfs                     1.8G         0      1.8G   0% /proc/timer_list
tmpfs                     1.8G         0      1.8G   0% /sys/firmware
/ # resize2fs /dev/sdb
resize2fs 1.43.3 (04-Sep-2016)
open: No such file or directory while opening /dev/sdb
/ # fdisk -l
/ #

Any ideas?

wminshew · 2018-08-24T18:23:13Z

Following up here: was able to solve the problem by re-launching the container with securityContext: privileged: true

fejta-bot · 2018-11-22T18:38:07Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.