Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support up to 8 EBS volumes. #66

Merged
merged 2 commits into from Sep 4, 2014
Merged

Support up to 8 EBS volumes. #66

merged 2 commits into from Sep 4, 2014

Conversation

rxin
Copy link
Member

@rxin rxin commented Sep 4, 2014

Please merge this at the same time as apache/spark#2260

@rxin rxin mentioned this pull request Sep 4, 2014
Closed
@shivaram
Copy link

shivaram commented Sep 4, 2014

Looks good. Is this or apache/spark#2260 in anyways not backwards compatible ?

@shivaram
Copy link

shivaram commented Sep 4, 2014

Also why do we need so many EBS devices ? Can't we just have a few large ones ?

@rxin
Copy link
Member Author

rxin commented Sep 4, 2014

Higher throughput. Each EBS volume is capped at around 130MBps.

@shivaram
Copy link

shivaram commented Sep 4, 2014

Ok - I see. So I thought about this a little more and this isn't backwards compatible as older spark_ec2 scripts will use /dev/sdv as the only volume.

To preserve backwards compatibility you can name them as sdv, sdw, sdx, sdy, sdz, sdu, sdt, sds, sdr ? (i.e. sdv, then 4 after and 4 before). Or maybe just symlinking /vol3 to /vol might be enough ?
Overall I'd suggest running spark_ec2.py from branch-1.1 with this spark-ec2 change and make sure it works.

@rxin
Copy link
Member Author

rxin commented Sep 4, 2014

Good idea. I symlinked vol3 to vol now.

I like this idea better than starting at sdv and then go around to sdu since that is more confusing.

@shivaram
Copy link

shivaram commented Sep 4, 2014

Cool. If you've tested this with the old spark_ec2.py, we can merge this.

@rxin
Copy link
Member Author

rxin commented Sep 4, 2014

Did you look at the apache spark one too? We should merge both at the same time.

@shivaram
Copy link

shivaram commented Sep 4, 2014

I took a brief look. I dont see why they need to go in at the same time ? This can go in before as its backward compatible ?

@rxin
Copy link
Member Author

rxin commented Sep 4, 2014

Oh true. Merging this please :)

@rxin
Copy link
Member Author

rxin commented Sep 4, 2014

Please also review the other one :)

rxin added a commit that referenced this pull request Sep 4, 2014
@rxin
Merge pull request #66 from rxin/v3-ebs
81bd75b 
Support up to 8 EBS volumes.
@rxin rxin merged commit 81bd75b into mesos:v3 Sep 4, 2014
asfgit pushed a commit to apache/spark that referenced this pull request Sep 5, 2014
@rxin
[SPARK-3391][EC2] Support attaching up to 8 EBS volumes.
1725a1a 
Please merge this at the same time as mesos/spark-ec2#66

Author: Reynold Xin <rxin@apache.org>

Closes #2260 from rxin/ec2-ebs-vol and squashes the following commits:

b9527d9 [Reynold Xin] Removed io1 ebs type.
bf9c403 [Reynold Xin] Made EBS volume type configurable.
c8e25ea [Reynold Xin] Support up to 8 EBS volumes.
adf4f2e [Reynold Xin] Revert git repo change.
020c542 [Reynold Xin] [SPARK-3391] Support attaching more than 1 EBS volumes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@rxin @shivaram