AWS Centos 6 AMIs do not resize the root partition #34

Closed
jeremiahsnapp opened this Issue Feb 19, 2016 · 8 comments

Comments

Projects
None yet
7 participants
@jeremiahsnapp

Problem

The AWS Centos 6 AMIs do not resize the root partition to use all of the attached EBS volume. This is a big problem because the default size is only 10GB which fills up very quickly especially when chef-server-ctl marketplace-setup ends up upgrading Chef packages.

The problem is that the AMI is relying on cloud-init's growpart module to resize the partition but apparently this only works for kernels > 3.8.

Reference: http://lists.openstack.org/pipermail/openstack/2014-August/008721.html

Growpart called by cloud-init only works for kernels >3.8. Only newer kernels support changing the partition size of a mounted partition. When using an older kernel the resizing of the root partition happens in the initrd stage before the root partition is mounted and the subsequent cloud-init growpart run is a no-op.

Here's the kernel in my Marketplace Chef Server I launched today.

[ec2-user@ip-172-31-5-119 ~]$ uname -r
2.6.32-573.12.1.el6.x86_64

Solutions

One solution would be to rebuild the Centos 6 AMIs using the old style of resizing which is described pretty well in the Mail List thread above and also in this blog post.

http://blog.backslasher.net/growroot-centos.html

After talking with @ryancragun though it seems that the better solution would be to build the AMIs using Centos 7 which is able to correctly use the cloud-init growpart module. Centos 7 build work is already underway so I think this is the direction @ryancragun plans to take to fix this issue.

Workaround

In the meantime you can use fdisk to resize the partition entry in the partition table while the partition is mounted and then reboot the instance. After the reboot finishes the entire EBS volume is completely available to the root partition without needing to run resize2fs. As long as you follow the steps below carefully resizing the partition can be done without losing any data.

The following is an example of the partition resizing procedure.

Be sure to login as the root user when running the following commands.

If you just launched the Marketplace Chef Server then you will want to wait approximately 15 minutes for the cloud-init work to finish running. Yes, it really takes that long. :)

You can run pgrep -lf S53cloud-final to see if the cloud-init process is still running.

Once cloud-init is finished you can run lsblk to see that the EBS volume attached as xvda has 30GB available but the xvda1 partition only has 10GB.

[root@ip-172-31-3-55 ~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  30G  0 disk
└─xvda1 202:1    0  10G  0 part /

First determine the Start sector of the xvda1 partition. Make sure you use sectors for units instead of the default of cylinders. The -u flag changes the units to sectors.

In this case the Start sector is 2000.

[root@ip-172-31-3-55 ~]# fdisk -lu /dev/xvda

Disk /dev/xvda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders, total 62914560 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0002370f

    Device Boot      Start         End      Blocks   Id  System
/dev/xvda1   *        2000    20971519    10484760   83  Linux

Now use the following command to set the units to sectors, delete the partition entry from the partition table and create a new partition entry using the same Start sector value, 2000 in this case. The empty line accepts the default End sector which will be the last sector of the disk. The w saves the modifications to the partition table.

[root@ip-172-31-3-55 ~]# fdisk /dev/xvda <<END
u
d
n
p
1
2000

p
w
END

Now reboot the instance and when you log back in you can run lsblk or df -h to see that the partition has been resized.

[root@ip-172-31-3-55 ~]# lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  30G  0 disk
└─xvda1 202:1    0  30G  0 part /
@Purple90

This comment has been minimized.

Show comment
Hide comment
@Purple90

Purple90 Apr 7, 2016

I can't see to ssh back into my aws instances once I run the commands. Did you have this issue?

Purple90 commented Apr 7, 2016

I can't see to ssh back into my aws instances once I run the commands. Did you have this issue?

@mihir-govil

This comment has been minimized.

Show comment
Hide comment
@mihir-govil

mihir-govil Apr 22, 2016

Hello Team,
LVM didn't create in the CentOS Linux release 7.2.1511 (Core) because of that we didn't able to extend the root partitions. Please share the details or steps how we extend the root partitions without LVM ?

[root@ip-******** ec2-user]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 9.8G 8.0G 1.3G 87% /

Hello Team,
LVM didn't create in the CentOS Linux release 7.2.1511 (Core) because of that we didn't able to extend the root partitions. Please share the details or steps how we extend the root partitions without LVM ?

[root@ip-******** ec2-user]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 9.8G 8.0G 1.3G 87% /

@Purple90

This comment has been minimized.

Show comment
Hide comment
@Purple90

Purple90 Apr 22, 2016

Hey mihir-govil, here is how I was able to solve it

Use 'lsblk' to verify that you do indeed have more space on your device than is showing in your root partition. If so, run the following command

"sudo growpart /dev/xvda 1"
"sudo reboot"

Hey mihir-govil, here is how I was able to solve it

Use 'lsblk' to verify that you do indeed have more space on your device than is showing in your root partition. If so, run the following command

"sudo growpart /dev/xvda 1"
"sudo reboot"

@emachnic

This comment has been minimized.

Show comment
Hide comment
@emachnic

emachnic Jun 24, 2016

I'd also like to add that you may need to install the cloud-utils-growpart package with yum to be able to do what @Purple90 said. Otherwise, his solution worked like a charm.

I'd also like to add that you may need to install the cloud-utils-growpart package with yum to be able to do what @Purple90 said. Otherwise, his solution worked like a charm.

@glasschef

This comment has been minimized.

Show comment
Hide comment
@glasschef

glasschef Jul 21, 2016

FYI, we're also seeing this issue in Centos 7: https://getchef.zendesk.com/agent/tickets/10386 (internal Chef ticket)

FYI, we're also seeing this issue in Centos 7: https://getchef.zendesk.com/agent/tickets/10386 (internal Chef ticket)

@jeremiahsnapp

This comment has been minimized.

Show comment
Hide comment
@jeremiahsnapp

jeremiahsnapp Sep 8, 2016

The newer Marketplace Chef Server built on Centos 7 doesn't auto resize its root partition because the cloud-utils-growpart package is not installed. I've tested this by using the cloud-boothook hook that cloud-init provides to install the cloud-utils-growpart package very early in the boot process. The package installs and cloud-init's growpart module automatically uses it to expand the partition.

As a workaround until cloud-utils-growpart gets built into the AMI you can put the following in your instance's user-data.

#cloud-boothook

#!/bin/bash

yum install -y cloud-utils-growpart

Reference: https://help.ubuntu.com/community/CloudInit

The newer Marketplace Chef Server built on Centos 7 doesn't auto resize its root partition because the cloud-utils-growpart package is not installed. I've tested this by using the cloud-boothook hook that cloud-init provides to install the cloud-utils-growpart package very early in the boot process. The package installs and cloud-init's growpart module automatically uses it to expand the partition.

As a workaround until cloud-utils-growpart gets built into the AMI you can put the following in your instance's user-data.

#cloud-boothook

#!/bin/bash

yum install -y cloud-utils-growpart

Reference: https://help.ubuntu.com/community/CloudInit

@ryancragun

This comment has been minimized.

Show comment
Hide comment
@ryancragun

ryancragun Sep 19, 2016

Contributor

The images published today contain growpart and expand as expected. Thanks for reporting it y'all.

Contributor

ryancragun commented Sep 19, 2016

The images published today contain growpart and expand as expected. Thanks for reporting it y'all.

@ryancragun ryancragun closed this Sep 19, 2016

@hilaby

This comment has been minimized.

Show comment
Hide comment
@hilaby

hilaby Apr 3, 2017

@Purple90 thanks, it worked:

After I typed sudo growpart /dev/xvda 1
This output came out CHANGED: partition=1 start=2048 old: size=62908492 end=62910540 new: size=209710462,end=209712510

Then I didn't have to reboot, I just typed sudo xfs_growfs -d / and it all just went smoothly.

hilaby commented Apr 3, 2017

@Purple90 thanks, it worked:

After I typed sudo growpart /dev/xvda 1
This output came out CHANGED: partition=1 start=2048 old: size=62908492 end=62910540 new: size=209710462,end=209712510

Then I didn't have to reboot, I just typed sudo xfs_growfs -d / and it all just went smoothly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment