Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for availability zones #72

Closed
coyotemarin opened this issue Dec 3, 2010 · 3 comments
Closed

add support for availability zones #72

coyotemarin opened this issue Dec 3, 2010 · 3 comments
Labels
Milestone

Comments

@coyotemarin
Copy link
Collaborator

We should add an availability_zone option to EMRJobRunner that lets you run job flows in specific availability zones (essentially, data centers).

@ghost ghost assigned coyotemarin Apr 11, 2011
@ghost ghost assigned irskep May 25, 2011
@irskep
Copy link
Contributor

irskep commented May 26, 2011

Should be done in my az_support branch (https://github.com/irskep/mrjob/tree/az_support) but I have not been able to fully test that it works. I can, however, confirm that it doesn't break anything.

Boto and Amazon appear to have a slightly different API for job descriptions. Amazon nests instance information (master/slave type, number, AZs, etc) in an InstanceConfig object called Instances, while Boto flattens it all out. I couldn't convince myself entirely that Boto actually works one way or the other, so I'm still testing that one by hand.

Edit: Job descriptions will most of the time have the 'availabilityzone' instance variable. Most of the time.

@irskep
Copy link
Contributor

irskep commented May 26, 2011

All day I have been asking myself the question, "Why is aws_availability_zone being ignored by no one except EMR?" The answer, it turns out, is that there is a bug in boto that sets the wrong key for the availability zone in the job flow configuration. So I will be sending them a pull request as well. The MRJob copy is fixed in my branch.

coyotemarin pushed a commit that referenced this issue May 27, 2011
Availability zone support (implements issue #72)
@coyotemarin
Copy link
Collaborator Author

Fixed in development-v0.2.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants