feat(opensearchservice): configuring gp3 throughput #26172

sktan · 2023-06-29T18:03:26Z

The interface EbsOptions for the opensearchservices CDK construct is missing a provisioned throughput option for eg gp3 instance types.
iops is there, but not throughput

Closes #26137.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

…an opensearch cluster

gitpod-io · 2023-06-29T18:03:45Z

corymhall · 2023-06-30T11:54:08Z

packages/aws-cdk-lib/aws-opensearchservice/lib/domain.ts

@@ -1575,6 +1583,7 @@ export class Domain extends DomainBase implements IDomain, ec2.IConnectable {
        volumeSize: ebsEnabled ? volumeSize : undefined,
        volumeType: ebsEnabled ? volumeType : undefined,
        iops: ebsEnabled ? props.ebs?.iops : undefined,
+        throughput: ebsEnabled ? props.ebs?.throughput : undefined,


It doesn't look like we have any check on the volumeType. It says iops and
throughput are only supported for iops/gp3 types, but we only check if it's ebs
type?

Do you know what happens if throughput is defined for gp2 volumes? Does it
just ignore that property or does it throw an error?

I think we need something like what we have in the volume construct

aws-cdk/packages/aws-cdk-lib/aws-ec2/lib/volume.ts

Lines 704 to 771 in 53d24f1

// Enforce maximum ratio of IOPS/GiB:

// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html

const maximumRatios: { [key: string]: number } = {};

maximumRatios[EbsDeviceVolumeType.GENERAL_PURPOSE_SSD_GP3] = 500;

maximumRatios[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD] = 50;

maximumRatios[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD_IO2] = 500;

const maximumRatio = maximumRatios[volumeType];

if (props.size && (props.iops > maximumRatio * props.size.toGibibytes({ rounding: SizeRoundingBehavior.FAIL }))) {

throw new Error(`\`${volumeType}\` volumes iops has a maximum ratio of ${maximumRatio} IOPS/GiB.`);

}

const maximumThroughputRatios: { [key: string]: number } = {};

maximumThroughputRatios[EbsDeviceVolumeType.GP3] = 0.25;

const maximumThroughputRatio = maximumThroughputRatios[volumeType];

if (props.throughput && props.iops) {

const iopsRatio = (props.throughput / props.iops);

if (iopsRatio > maximumThroughputRatio) {

throw new Error(`Throughput (MiBps) to iops ratio of ${iopsRatio} is too high; maximum is ${maximumThroughputRatio} MiBps per iops`);

}

}

}

if (props.enableMultiAttach) {

const volumeType = props.volumeType ?? EbsDeviceVolumeType.GENERAL_PURPOSE_SSD;

if (

![

EbsDeviceVolumeType.PROVISIONED_IOPS_SSD,

EbsDeviceVolumeType.PROVISIONED_IOPS_SSD_IO2,

].includes(volumeType)

) {

throw new Error('multi-attach is supported exclusively on `PROVISIONED_IOPS_SSD` and `PROVISIONED_IOPS_SSD_IO2` volumes.');

}

}

if (props.size) {

const size = props.size.toGibibytes({ rounding: SizeRoundingBehavior.FAIL });

// Enforce minimum & maximum volume size:

// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-ebs-volume.html

const sizeRanges: { [key: string]: { Min: number, Max: number } } = {};

sizeRanges[EbsDeviceVolumeType.GENERAL_PURPOSE_SSD] = { Min: 1, Max: 16384 };

sizeRanges[EbsDeviceVolumeType.GENERAL_PURPOSE_SSD_GP3] = { Min: 1, Max: 16384 };

sizeRanges[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD] = { Min: 4, Max: 16384 };

sizeRanges[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD_IO2] = { Min: 4, Max: 16384 };

sizeRanges[EbsDeviceVolumeType.THROUGHPUT_OPTIMIZED_HDD] = { Min: 125, Max: 16384 };

sizeRanges[EbsDeviceVolumeType.COLD_HDD] = { Min: 125, Max: 16384 };

sizeRanges[EbsDeviceVolumeType.MAGNETIC] = { Min: 1, Max: 1024 };

const volumeType = props.volumeType ?? EbsDeviceVolumeType.GENERAL_PURPOSE_SSD;

const { Min, Max } = sizeRanges[volumeType];

if (size < Min || size > Max) {

throw new Error(`\`${volumeType}\` volumes must be between ${Min} GiB and ${Max} GiB in size.`);

}

}

if (props.throughput) {

const throughputRange = { Min: 125, Max: 1000 };

const { Min, Max } = throughputRange;

if (props.volumeType != EbsDeviceVolumeType.GP3) {

throw new Error(

'throughput property requires volumeType: EbsDeviceVolumeType.GP3',

);

}

if (props.throughput < Min || props.throughput > Max) {

throw new Error(

`throughput property takes a minimum of ${Min} and a maximum of ${Max}`,

);

}

}

I did think about checks specification checks, the other option for iops didn't any checks defined so I didn't it for the throughput option`. I'll get that sorted in the next commit :)

Added a bunch of tests and validation checks for you @corymhall , let me know if there's anything i've missed.

Pull request has been modified.

aws-cdk-automation · 2023-07-18T00:11:17Z

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

jumic · 2023-07-24T21:03:08Z

@sktan The last build failed. Do you need help?
If the build is successful locally and only failed in this PR: Merge branch main into your branch to trigger a new build.

sktan · 2023-07-25T02:31:47Z

@sktan The last build failed. Do you need help? If the build is successful locally and only failed in this PR: Merge branch main into your branch to trigger a new build.

Hey, thanks for the offer @jumic. I haven't gotten around to fixing it just yet, it failed due to another PR making a change to the default parameters causing my integration test to provide a CHANGED result.

This will get sorted soon

aws-cdk-automation · 2023-07-28T00:08:21Z

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

aws-cdk-automation · 2023-07-29T00:08:24Z

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

aws-cdk-automation · 2023-07-30T00:09:04Z

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

aws-cdk-automation · 2023-07-31T00:08:42Z

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

sktan · 2023-08-07T02:11:46Z

@corymhall we should be good to go again for this PR

jogold · 2023-08-18T13:09:30Z

packages/aws-cdk-lib/aws-opensearchservice/lib/domain.ts

+    // https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-opensearchservice-domain-ebsoptions.html
+    if (ebsEnabled) {
+      // Check if iops or throughput if general purpose is configured
+      if (volumeType == ec2.EbsDeviceVolumeType.GENERAL_PURPOSE_SSD || volumeType == ec2.EbsDeviceVolumeType.STANDARD) {


Suggested change

if (volumeType == ec2.EbsDeviceVolumeType.GENERAL_PURPOSE_SSD || volumeType == ec2.EbsDeviceVolumeType.STANDARD) {

if (volumeType === ec2.EbsDeviceVolumeType.GENERAL_PURPOSE_SSD || volumeType === ec2.EbsDeviceVolumeType.STANDARD) {

mergify · 2023-08-22T12:18:28Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Pull request has been modified.

mergify · 2023-08-22T13:24:23Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

aws-cdk-automation · 2023-08-22T14:44:29Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: d51f153
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2023-08-22T14:44:55Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

sktan added 5 commits June 29, 2023 17:54

Added support for providing throughput as an option when configuring …

60a0504

…an opensearch cluster

Integration test for opensearch gp3

5d1cf0a

Added integration test snapshot

648ef49

Updated README to include instructions on using gp3 volumes

51eb22f

Changed test to use ENUM rather than string resolver

cab621a

sktan changed the title ~~feat(aws-opensearchservice): Support gp3 throughput configuration when configuring an opensearch domain~~ feat(opensearchservice): Support gp3 throughput configuration when configuring an opensearch domain Jun 29, 2023

Merge branch 'main' into feat/aws-opensearchservice/ebsoptionsthroughput

d70c7e2

github-actions bot added effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2 beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK labels Jun 29, 2023

Removed trailing space

a0d9898

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Jun 29, 2023

aws-cdk-automation requested a review from a team June 29, 2023 19:40

corymhall previously requested changes Jun 30, 2023

View reviewed changes

corymhall removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Jun 30, 2023

Added some EBS Options validation checks and associated unit tests

4632845

sktan added 4 commits July 4, 2023 19:23

fixed linter issues

56c6261

another lint fix

6e6f398

fixed tests

2681b55

Forgot to include the changed library

81419f3

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Jul 5, 2023

sktan requested a review from corymhall July 5, 2023 01:51

Merge branch 'main' into feat/aws-opensearchservice/ebsoptionsthroughput

15c79cb

github-actions bot mentioned this pull request Jul 26, 2023

Monthly PRs metrics report - June 2023 #26516

Closed

sktan and others added 4 commits July 31, 2023 10:09

Merge branch 'main' into feat/aws-opensearchservice/ebsoptionsthroughput

c2bda1b

Updated tests after merge conflict

43553b3

Removed unnessecary test file

a223215

Merge branch 'main' into feat/aws-opensearchservice/ebsoptionsthroughput

7446465

jogold reviewed Aug 18, 2023

View reviewed changes

aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Aug 18, 2023

rix0rrr changed the title ~~feat(opensearchservice): Support gp3 throughput configuration when configuring an opensearch domain~~ feat(opensearchservice): configuring gp3 throughput Aug 22, 2023

rix0rrr previously approved these changes Aug 22, 2023

View reviewed changes

Merge branch 'main' into feat/aws-opensearchservice/ebsoptionsthroughput

936cd25

rix0rrr approved these changes Aug 22, 2023

View reviewed changes

Merge branch 'main' into feat/aws-opensearchservice/ebsoptionsthroughput

d51f153

mergify bot merged commit 47df704 into aws:main Aug 22, 2023
9 checks passed

sktan deleted the feat/aws-opensearchservice/ebsoptionsthroughput branch August 23, 2023 00:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(opensearchservice): configuring gp3 throughput #26172

feat(opensearchservice): configuring gp3 throughput #26172

sktan commented Jun 29, 2023 •

edited by rix0rrr

gitpod-io bot commented Jun 29, 2023

corymhall Jun 30, 2023

sktan Jul 4, 2023

sktan Jul 5, 2023 •

edited

aws-cdk-automation commented Jul 18, 2023

jumic commented Jul 24, 2023

sktan commented Jul 25, 2023

aws-cdk-automation commented Jul 28, 2023

aws-cdk-automation commented Jul 29, 2023

aws-cdk-automation commented Jul 30, 2023

aws-cdk-automation commented Jul 31, 2023

sktan commented Aug 7, 2023

jogold Aug 18, 2023

mergify bot commented Aug 22, 2023

mergify bot commented Aug 22, 2023

aws-cdk-automation commented Aug 22, 2023

mergify bot commented Aug 22, 2023

	// Enforce maximum ratio of IOPS/GiB:
	// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html
	const maximumRatios: { [key: string]: number } = {};
	maximumRatios[EbsDeviceVolumeType.GENERAL_PURPOSE_SSD_GP3] = 500;
	maximumRatios[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD] = 50;
	maximumRatios[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD_IO2] = 500;
	const maximumRatio = maximumRatios[volumeType];
	if (props.size && (props.iops > maximumRatio * props.size.toGibibytes({ rounding: SizeRoundingBehavior.FAIL }))) {
	throw new Error(`\`${volumeType}\` volumes iops has a maximum ratio of ${maximumRatio} IOPS/GiB.`);
	}

	const maximumThroughputRatios: { [key: string]: number } = {};
	maximumThroughputRatios[EbsDeviceVolumeType.GP3] = 0.25;
	const maximumThroughputRatio = maximumThroughputRatios[volumeType];
	if (props.throughput && props.iops) {
	const iopsRatio = (props.throughput / props.iops);
	if (iopsRatio > maximumThroughputRatio) {
	throw new Error(`Throughput (MiBps) to iops ratio of ${iopsRatio} is too high; maximum is ${maximumThroughputRatio} MiBps per iops`);
	}

	}
	}

	if (props.enableMultiAttach) {
	const volumeType = props.volumeType ?? EbsDeviceVolumeType.GENERAL_PURPOSE_SSD;
	if (
	![
	EbsDeviceVolumeType.PROVISIONED_IOPS_SSD,
	EbsDeviceVolumeType.PROVISIONED_IOPS_SSD_IO2,
	].includes(volumeType)
	) {
	throw new Error('multi-attach is supported exclusively on `PROVISIONED_IOPS_SSD` and `PROVISIONED_IOPS_SSD_IO2` volumes.');
	}
	}

	if (props.size) {
	const size = props.size.toGibibytes({ rounding: SizeRoundingBehavior.FAIL });
	// Enforce minimum & maximum volume size:
	// https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-ebs-volume.html
	const sizeRanges: { [key: string]: { Min: number, Max: number } } = {};
	sizeRanges[EbsDeviceVolumeType.GENERAL_PURPOSE_SSD] = { Min: 1, Max: 16384 };
	sizeRanges[EbsDeviceVolumeType.GENERAL_PURPOSE_SSD_GP3] = { Min: 1, Max: 16384 };
	sizeRanges[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD] = { Min: 4, Max: 16384 };
	sizeRanges[EbsDeviceVolumeType.PROVISIONED_IOPS_SSD_IO2] = { Min: 4, Max: 16384 };
	sizeRanges[EbsDeviceVolumeType.THROUGHPUT_OPTIMIZED_HDD] = { Min: 125, Max: 16384 };
	sizeRanges[EbsDeviceVolumeType.COLD_HDD] = { Min: 125, Max: 16384 };
	sizeRanges[EbsDeviceVolumeType.MAGNETIC] = { Min: 1, Max: 1024 };
	const volumeType = props.volumeType ?? EbsDeviceVolumeType.GENERAL_PURPOSE_SSD;
	const { Min, Max } = sizeRanges[volumeType];
	if (size < Min \|\| size > Max) {
	throw new Error(`\`${volumeType}\` volumes must be between ${Min} GiB and ${Max} GiB in size.`);
	}
	}

	if (props.throughput) {
	const throughputRange = { Min: 125, Max: 1000 };
	const { Min, Max } = throughputRange;
	if (props.volumeType != EbsDeviceVolumeType.GP3) {
	throw new Error(
	'throughput property requires volumeType: EbsDeviceVolumeType.GP3',
	);
	}
	if (props.throughput < Min \|\| props.throughput > Max) {
	throw new Error(
	`throughput property takes a minimum of ${Min} and a maximum of ${Max}`,
	);
	}
	}

	if (volumeType == ec2.EbsDeviceVolumeType.GENERAL_PURPOSE_SSD \|\| volumeType == ec2.EbsDeviceVolumeType.STANDARD) {
	if (volumeType === ec2.EbsDeviceVolumeType.GENERAL_PURPOSE_SSD \|\| volumeType === ec2.EbsDeviceVolumeType.STANDARD) {

feat(opensearchservice): configuring gp3 throughput #26172

feat(opensearchservice): configuring gp3 throughput #26172

Conversation

sktan commented Jun 29, 2023 • edited by rix0rrr

gitpod-io bot commented Jun 29, 2023

corymhall Jun 30, 2023

Choose a reason for hiding this comment

sktan Jul 4, 2023

Choose a reason for hiding this comment

sktan Jul 5, 2023 • edited

Choose a reason for hiding this comment

aws-cdk-automation commented Jul 18, 2023

jumic commented Jul 24, 2023

sktan commented Jul 25, 2023

aws-cdk-automation commented Jul 28, 2023

aws-cdk-automation commented Jul 29, 2023

aws-cdk-automation commented Jul 30, 2023

aws-cdk-automation commented Jul 31, 2023

sktan commented Aug 7, 2023

jogold Aug 18, 2023

Choose a reason for hiding this comment

mergify bot commented Aug 22, 2023

mergify bot commented Aug 22, 2023

aws-cdk-automation commented Aug 22, 2023

AWS CodeBuild CI Report

mergify bot commented Aug 22, 2023

sktan commented Jun 29, 2023 •

edited by rix0rrr

sktan Jul 5, 2023 •

edited