update of master_root_volume_size

[wjaskowski]

I noticed that while you can set the new value of compute_root_volume_size and new nodes will have the desired volume, setting master_root_volume_size and updating the cluster does not make any effect. I also tried to restart the master node but without luck. Is there any way to update the size of the master root volume?

[fnubalaj]

I don't think that is currently possible. Because the volume is created when the chef recipes are built which is done only once on cluster creation for the master volume. For the compute nodes, it is done each time a compute node comes online. I will keep investigating to see if there is another way this could be achieved and also file an enhancement for this feature.

[bwbarrett]

There is not a way through CfnCluster. CfnCluster's mechanism for modifying instance parameters is the AWS CloudFormation stack that it creates when you create the cluster. CloudFormation's solution to the change would be to recreate the instance, which isn't what you want.

However, the master root volume is just an ordinary EBS volume. If you follow https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-modify-volume.html, you should be able to resize the root volume. You are likely to have more luck if the base os is Amazon Linux or Ubuntu 16.04, but it should work with any Linux distro that the instructions say is supported. Note that this process isn't entirely risk free; changing the size of a live filesystem is as tricky as it sounds.

[fnubalaj]

Just to correct what I formerly said, chef recipes do not control the root_volume_size. It is controlled in the cloud formation template and any change to an block device mappings of an instance needs to launch a replacement instance.

See: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#cfn-ec2-instance-blockdevicemappings

[wjaskowski]

I see. Shouldn't then the cfncluster warn me that this setting has changed but won't be updated? It just silently did the update.

[enrico-usai]

This issue has been addressed as part of the #1583

With this patch, the process will check for the updatablity of each detected change and will prevent the update from being run if any change is marked as not safe or not possible.

[enrico-usai]

Release 2.8.0 is out (and 2.8.1 too). One of the enhancements of the 2.8.0 is: "Renewed user experience and robustness of cluster update functionality" and this failure during the update will be prevented.

I'm going to close this issue, feel free to open a new one if you encounter similar problems.