AWS EBS gp3 and io2 Support for etcd Volumes

[msidwell]

1. Describe IN DETAIL the feature/behavior/change you would like to see.
Add support to addAWSVolume() in master_volumes.go for etcd volumes to use AWS gp3 SSD's as they are cheaper and more performant than gp2. Adding support for etcd io2 volumes is also desired due to their higher durability objective relative to io1.

(Edited to remove gp3 node root volume support and defaults which is covered by #10345)

2. Feel free to provide a design supporting your feature request.
The etcd volumeType parameter does not respect entering "gp3" or "io2" due to the switch/case logic built into it:

func (b *MasterVolumeBuilder) addAWSVolume(c *fi.ModelBuilderContext, name string, volumeSize int32, zone string, etcd kops.EtcdClusterSpec, m kops.EtcdMemberSpec, allMembers []string) error {
	volumeType := fi.StringValue(m.VolumeType)
	volumeIops := fi.Int32Value(m.VolumeIops)
	switch volumeType {
	case "io1":
		if volumeIops <= 0 {
			volumeIops = DefaultAWSEtcdVolumeIops
		}
	default:
		volumeType = DefaultAWSEtcdVolumeType
	}

Adding cases for "io2" and "gp3" should allow for their consumption through this volume builder function. io2 volumes have the same minimum IOPS parameter as io1, so the if volumeIops <= 0 check can be applied there as well. As part of this addition, that logic should check for if volumeIops < 100 to align with the actual minimums of these io* volumes. For gp3 volumes, 3,000 IOPS and 125 MiBps are provided with any size disk, so there should be validation logic that defaults to those minimums if entered parameters fall below them. Part of adding gp3 volume support would ideally include the addition of a volumeThroughput parameter to cover all configurable components of gp3:

const (
	DefaultEtcdVolumeSize    = 20
	DefaultAWSEtcdVolumeType = "gp2"
	DefaultAWSEtcdVolumeIops = 100
        DefaultAWSEtcdVolumeGp3Iops = 3000
        DefaultAWSEtcdVolumeGp3Throughput = 125
	DefaultGCEEtcdVolumeType = "pd-ssd"
	DefaultALIEtcdVolumeType = "cloud_ssd"
)
...
func (b *MasterVolumeBuilder) addAWSVolume(c *fi.ModelBuilderContext, name string, volumeSize int32, zone string, etcd kops.EtcdClusterSpec, m kops.EtcdMemberSpec, allMembers []string) error {
	volumeType := fi.StringValue(m.VolumeType)
	volumeIops := fi.Int32Value(m.VolumeIops)
       
	switch volumeType {
	case "io1":
		if volumeIops < 100 {
			volumeIops = DefaultAWSEtcdVolumeIops
		}
        case "io2":
		if volumeIops < 100 {
			volumeIops = DefaultAWSEtcdVolumeIops
		}
        case "gp3":
		if volumeIops < 3000 {
			volumeIops = DefaultAWSEtcdVolumeGp3Iops
		}
               if volumeThroughput < 125 {
                       volumeThroughput = DefaultAWSEtcdVolumeGp3Throughput
               }
	default:
		volumeType = DefaultAWSEtcdVolumeType
	}

For io2 parameter validation, the maximum ratio of provisioned IOPS to requested volume size (in GiB) is 500:1 versus io1's 50:1. As for gp3, the maximum ratio of provisioned IOPS to provisioned volume size is 500 IOPS per GiB and the maximum ratio of provisioned throughput to provisioned IOPS is .25 MiB/s per IOPS. The following volume configurations support provisioning either maximum IOPS or maximum throughput:

32 GiB or larger: 500 IOPS/GiB x 32 GiB = 16,000 IOPS
8 GiB or larger and 4,000 IOPS or higher: 4,000 IOPS x 0.25 MiB/s/IOPS = 1,000 MiB/s

This can be implemented in validation logic like so:

if strings.Contains(volumeType, "io") || volumeType == "gp3" {
    t.VolumeIops = i64(int64(volumeIops))
    if volumeType == "io1" {
        // https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
        if float64(*t.VolumeIops)/float64(*t.SizeGB) > 50.0 {
	    return fmt.Errorf("volumeIops to volumeSize ratio must be lower than 50. For %s ratio is %f", *t.Name, float64(*t.VolumeIops)/float64(*t.SizeGB))
	}
    } else {
        if float64(*t.VolumeIops)/float64(*t.SizeGB) > 500.0 {
	    return fmt.Errorf("volumeIops to volumeSize ratio must be lower than 500. For %s ratio is %f", *t.Name, float64(*t.VolumeIops)/float64(*t.SizeGB))
    }
   if volumeType == "gp3" {
        t.VolumeThroughput = i64(int64(volumeThroughput))
        if float64(*t.VolumeThroughput)/float64(*t.VolumeIops) > 0.25 {
            return fmt.Errorf("volumeThroughput to volumeIops ratio must be lower than0.25. For %s ratio is %f", *t.Name, float64(*t.VolumeThroughput)/float64(*t.VolumeIops))
        }
   }
}

[olemarkus]

Thanks for all this. No chance you can submit a PR? Seems like you have a good understanding of what should be added.

[msidwell]

Can do, I'll attach it in this request thread once I've submitted it