Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stepfunctions-tasks: emr-create-cluster - add support for managedScalingPolicy #26115

Open
2 tasks
kshefchek opened this issue Jun 26, 2023 · 3 comments
Open
2 tasks
Labels
@aws-cdk/aws-emr Related to Amazon EMR @aws-cdk/aws-stepfunctions-tasks effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2

Comments

@kshefchek
Copy link

Describe the feature

When creating an EMR cluster in a step function it's possible to create a managed scaling policy when authoring amazon states language json. However, there's no way to do this in cdk that I can see. Would it be possible to add managed scaling policy support in cdk?

Use Case

I want to create an EMR cluster with emr managed scaling in a step function with cdk.

Proposed Solution

No response

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.55.1

Environment details (OS name and version, etc.)

macos

@kshefchek kshefchek added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jun 26, 2023
@github-actions github-actions bot added the @aws-cdk/aws-emr Related to Amazon EMR label Jun 26, 2023
@pahud pahud changed the title aws-stepfunctions-tasks/emr-create-cluster: add support for managedScalingPolicy stepfunctions-tasks: emr-create-cluster - add support for managedScalingPolicy Jun 26, 2023
@pahud
Copy link
Contributor

pahud commented Jun 26, 2023

When creating an EMR cluster in a step function it's possible to create a managed scaling policy when authoring amazon states language json

Can you share any links to the aws doc about this? I think it makes perfect sense to support that in aws-stepfunctions-tasks but we just need to know how to configure that correctly with the parameters.

If you have any working sample that might be helpful, feel free to share the code snippets here. Thanks.

@pahud pahud added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Jun 26, 2023
@kshefchek
Copy link
Author

I could not find any official documentation for this in the step function docs, but it is part of the RunJobFlow API documented here: https://docs.aws.amazon.com/emr/latest/APIReference/API_RunJobFlow.html, ie

...
"LogUri": "string",
"ManagedScalingPolicy": { 
      "ComputeLimits": { 
         "MaximumCapacityUnits": number,
         "MaximumCoreCapacityUnits": number,
         "MaximumOnDemandCapacityUnits": number,
         "MinimumCapacityUnits": number,
         "UnitType": "string"
      }
},
"Name": "string",
...

There is an example here - https://medium.com/@amit.kumarbasu/launch-an-emr-managed-scaling-cluster-with-step-functions-bc544babc80f

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jun 26, 2023
@valentinlagunes
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-emr Related to Amazon EMR @aws-cdk/aws-stepfunctions-tasks effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2
Projects
None yet
Development

No branches or pull requests

3 participants