Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Utilize InstanceFleet features in EMR Cluster Mgmt #328

Open
nateiam opened this issue May 13, 2017 · 0 comments
Open

Utilize InstanceFleet features in EMR Cluster Mgmt #328

nateiam opened this issue May 13, 2017 · 0 comments
Milestone
Release 0.42.0

Comments

@nateiam
Copy link
Contributor

nateiam commented May 13, 2017

As a Herd/DM Cluster Mgmt User I want to use new EMR InstanceFleet features so I can manage spot instances more effectively.

Teams want to take advantage of the following features: Instance Fleets with Weighted Capacity, Defined Duration, Provisioning Timeout as documented in this AWS [documentation|http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-instance-fleet.html]

This story adds an InstanceFleet section to Herd's existing [EMR Cluster Definition|https://wiki.finra.org/display/DataManagement/EMR+Cluster+Definition]

Acceptance Criteria

  • New optional element exists in Cluster Definition for InstanceFleet data
    • InstanceFleet data includes all information in the instance-fleet json as described in the AWS documentation linked above
  • All InstanceFleet information gets passed to AWS when:
    • User includes InstanceFleet infromation in a Cluster Definition
    • User includes InstanceFleet infromation as an override when launching a cluster
  • Existing EMR Cluster Activiti wrapper is modified to handle InstanceFleet
  • Return 40x error if user includes both InstanceFleets and InstanceDefinitions information
    • If override creates a definition that includes both InstanceFleet and pre-existing InstanceDefinitions elements, return 40x error stating they must include only one
  • Return 40x if EMR version does not support Instance Fleet

Test Notes

  • Proposed testing approach
    • Validate all inputs get passed according to AWS spec when launching EMR
    • New smoke test for happy path - but it has to be something that is predictable/reliable
@nateiam nateiam added this to the Release 0.42.0 milestone May 13, 2017
FINRAOSS pushed a commit that referenced this issue May 18, 2018
Merge pull request #328 in DM/herd from develop to master
5e8861f 
* commit '955aaae3615ceb505424379bb26af1db3aaa4d5c':
  Added initial version of the herd.postgres.0.69.0-to-0.70.0.upgrade.sql script
  Adding empty file placeholder to avoid Git EOL Style error
  Added initial version of the herd.postgres.0.69.0-to-0.70.0.upgrade.sql script
  DM-10718: Factory CQ - Optimize a poorly performing query that occurs during a BData POST
  DM-10707 Suggest an edit to BDef Descriptive Information
  DM-10707 Suggest an edit to BDef Descriptive Information
  DM-10690: Factory CQ - NullPointerException thrown in herd IndexFunctionsDaoImpl.buildIndexSearchResults()
  DM-10629: Factory CQ - NullPointerException thrown in herd IndexFunctionsDaoImpl.getIndexStats()
  DM-10629: Factory CQ - NullPointerException thrown in herd IndexFunctionsDaoImpl.getIndexStats()
  DM-10707 Suggest an edit to BDef Descriptive Information
  DM-10684: Locate BData eligible for retention expiration based on BData-level date
  Increment Maven POM version numbers to 0.70.0-SNAPSHOT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Release 0.42.0
Development

No branches or pull requests
1 participant
@nateiam