Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

terminate idle job flows near end of hour #319

Closed
coyotemarin opened this issue Dec 21, 2011 · 2 comments · Fixed by #334
Closed

terminate idle job flows near end of hour #319

coyotemarin opened this issue Dec 21, 2011 · 2 comments · Fixed by #334
Labels
Feature
Milestone
v0.3.2

Comments

@coyotemarin
Copy link
Collaborator

Since Amazon bills by the full hour, it'd be nice if mrjob.tools.emr.terminate_idle_job_flows could terminate job flows that are about to start a new hour. For example if terminate_idle_job_flows was cronned to run every 5 minutes, you could tell it to terminate job flows that are 6 minutes or less from starting a new hour.

The pricing model is actually a little more complicated than this, and this doesn't work well at all if you, say, added more nodes a half hour after starting your job flow, but it's probably useful for most cases.
@coyotemarin
Copy link
Collaborator Author

Moving this to v0.3.2. We've been using pooling in dev, and we end up with a lot of waste at the end of job flows.

Probably would make sense to only do this for pooled job flows.

@coyotemarin coyotemarin mentioned this issue Jan 12, 2012
Merged
@irskep
Copy link
Contributor

irskep commented Jan 12, 2012

Fixed by #334.

@irskep irskep closed this as completed Jan 12, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature
Projects
None yet
Milestone
v0.3.2
Development

Successfully merging a pull request may close this issue.

Terminate at end of hour coyotemarin/mrjob
2 participants
@irskep @coyotemarin