Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(stepfunctions-tasks): add validations for EmrCreateCluster #28529

Merged
merged 4 commits into from
Dec 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
kaizencc marked this conversation as resolved.
Show resolved Hide resolved

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Large diffs are not rendered by default.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "3c88aacdb6b48767fb52367e6dc0fe01602cfdc730dee4c4e3bebe0cec85ff9e.zip"
"S3Key": "eed45a32a57f32bc36031539054db0b27239d161061c528482bb55be51068664.zip"
},
"Environment": {
"Variables": {
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -582,13 +582,17 @@ export namespace EmrCreateCluster {
/**
* The bid price for each EC2 Spot instance type as defined by InstanceType. Expressed in USD.
*
* Cannot specify both `bidPrice` and `bidPriceAsPercentageOfOnDemandPrice`.
*
* @default - None
*/
readonly bidPrice?: string;

/**
* The bid price, as a percentage of On-Demand price.
*
* Cannot specify both `bidPrice` and `bidPriceAsPercentageOfOnDemandPrice`.
*
* @default - None
*/
readonly bidPriceAsPercentageOfOnDemandPrice?: number;
Expand Down Expand Up @@ -713,6 +717,8 @@ export namespace EmrCreateCluster {

/**
* The spot provisioning timeout period in minutes.
*
* The value must be between 5 and 1440.
*/
readonly timeoutDurationMinutes: number;
}
Expand Down Expand Up @@ -781,14 +787,26 @@ export namespace EmrCreateCluster {
/**
* The target capacity of On-Demand units for the instance fleet, which determines how many On-Demand instances to provision.
*
* If not specified or set to 0, only Spot Instances are provisioned for the instance fleet using `targetSpotCapacity`.
*
* At least one of `targetSpotCapacity` and `targetOnDemandCapacity` should be greater than 0.
* For a master instance fleet, only one of `targetSpotCapacity` and `targetOnDemandCapacity` can be specified, and its value
* must be 1.
*
* @default No targetOnDemandCapacity
*/
readonly targetOnDemandCapacity?: number;

/**
* The target capacity of Spot units for the instance fleet, which determines how many Spot instances to provision
*
* @default No targetSpotCapacity
* If not specified or set to 0, only On-Demand Instances are provisioned for the instance fleet using `targetOnDemandCapacity`.
*
* At least one of `targetSpotCapacity` and `targetOnDemandCapacity` should be greater than 0.
* For a master instance fleet, only one of `targetSpotCapacity` and `targetOnDemandCapacity` can be specified, and its value
* must be 1.
*
* @default No targetSpotCapacity
*/
readonly targetSpotCapacity?: number;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,10 @@ export function EbsConfigurationPropertyToJson(property: EmrCreateCluster.EbsCon
* @param property
*/
export function InstanceTypeConfigPropertyToJson(property: EmrCreateCluster.InstanceTypeConfigProperty) {
if (property.bidPrice && property.bidPriceAsPercentageOfOnDemandPrice) {
throw new Error('Cannot specify both bidPrice and bidPriceAsPercentageOfOnDemandPrice');
}

return {
BidPrice: cdk.stringToCloudFormation(property.bidPrice),
BidPriceAsPercentageOfOnDemandPrice: cdk.numberToCloudFormation(property.bidPriceAsPercentageOfOnDemandPrice),
Expand Down Expand Up @@ -146,6 +150,9 @@ function SpotProvisioningSpecificationPropertyToJson(property?: EmrCreateCluster
if (!property) {
return undefined;
}
if (!cdk.Token.isUnresolved(property.timeoutDurationMinutes) && (property.timeoutDurationMinutes < 5 || property.timeoutDurationMinutes > 1440)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to your PR, but I can't believe we have a prop called timeoutDurationMinutes: number. It should have been typed and named as timeout: Duration. Wonder if you're willing to deprecate timeoutDurationMinutes in favor of a new prop in a separate PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaizencc

Wonder if you're willing to deprecate timeoutDurationMinutes in favor of a new prop in a separate PR

OK, I will take it.

Copy link
Contributor Author

@go-to-k go-to-k Dec 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaizencc

Oh, this param timeoutDurationMinutes is required in the interface. This means that even if we have a new parameter, we still need to specify this parameter, and making it optional would be a breaking change.

I was mistaken: making a required parameter optional for an interface that is a props is not a breaking change.

But making the new parameter require is a breaking change.

What should we do? Should it be added as a new optional parameter, whereas before it was required? Or do we keep going as is?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll submit a PR anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaizencc

I have submitted the PR.

#28532 (comment)

throw new Error(`timeoutDurationMinutes must be between 5 and 1440, got ${property.timeoutDurationMinutes}`);
}
return {
AllocationStrategy: cdk.stringToCloudFormation(property.allocationStrategy),
BlockDurationMinutes: cdk.numberToCloudFormation(property.blockDurationMinutes),
Expand All @@ -160,6 +167,20 @@ function SpotProvisioningSpecificationPropertyToJson(property?: EmrCreateCluster
* @param property
*/
export function InstanceFleetConfigPropertyToJson(property: EmrCreateCluster.InstanceFleetConfigProperty) {
if (!property.targetSpotCapacity && !property.targetOnDemandCapacity) {
throw new Error('At least one of targetSpotCapacity and targetOnDemandCapacity should be greater than 0');
}
if (property.instanceFleetType === EmrCreateCluster.InstanceRoleType.MASTER) {
if (property.targetSpotCapacity && property.targetOnDemandCapacity) {
throw new Error('For a master instance fleet, only one of targetSpotCapacity and targetOnDemandCapacity can be specified');
}
if (property.targetSpotCapacity && property.targetSpotCapacity !== 1) {
throw new Error(`For a master instance fleet, targetSpotCapacity cannot be a number other than 1, got ${property.targetSpotCapacity}`);
}
if (property.targetOnDemandCapacity && property.targetOnDemandCapacity !== 1) {
throw new Error(`For a master instance fleet, targetOnDemandCapacity cannot be a number other than 1, got ${property.targetOnDemandCapacity}`);
}
}
return {
InstanceFleetType: cdk.stringToCloudFormation(property.instanceFleetType?.valueOf()),
InstanceTypeConfigs: cdk.listMapper(InstanceTypeConfigPropertyToJson)(property.instanceTypeConfigs),
Expand Down
Loading
Loading