[SPARK-9314] [EC2] add root EBS config options to spark-ec2

[kmaehashi]

This fixes [SPARK-9314].

This patch adds two options to spark-ec2 command:

• --ebs-root-vol-size to specify the custom size of the root EBS volume in GiB.
• --ebs-root-vol-type to specify the custom type of the root EBS volume (gp2, standard, etc.)

When these options are not specified, the default value from AMI is used.

[AmplabJenkins]

Can one of the admins verify this patch?

[kmaehashi]

@shivaram @nchammas Could you please confirm this patch?

[shivaram]

@kmaehashi Does this work in practice ? AFAIK the AMI dictates the size of the root volume and one needs to do a copy + resize to expand it ? See [1]

[1] http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html

[kmaehashi]

@shivaram Yes, you can do that via the script specified in --user-data.

[shivaram]

Not sure I get it -- Does this mean the increased size will only work if we use the resizefs in a script ?

[felixmaximilian]

What is the current state of this fix? I got trouble with the standard root dir size when building my custom spark. This pr should fix this!

[felixmaximilian]

I tried this with an instance type where the /root dir isn't on an EBS. There the volume size parameter hasn't had an effect. It might be a good extension to specify the /root volume size independently of the storage used for it.

[andrewor14]

What's the status of this patch? @kmaehashi have you had the chance to address the comments? If you're not actively working on it then we should close it for now and re-open it later if there is interest.

[kmaehashi]

Even if you use the option --ebs-root-vol-size proposed in this PR, you still need to expand partition size (using parted) and file system size (using resize2fs) by yourself, as documented in AWS user guide. Of course you can automate these procedures via --user-data.

http://docs.aws.amazon.com/en_us/AWSEC2/latest/UserGuide/storage_expand_partition.html
http://docs.aws.amazon.com/en_us/AWSEC2/latest/UserGuide/ebs-expand-volume.html#recognize-expanded-volume-linux

As --ebs-root-vol-size option is useful in my use case, it was a bit confusing for general users who expect that root volume size to be increased automatically. So I'm going to remove --ebs-root-vol-size option in this PR.

[nchammas]

Perhaps I misunderstood you, but I think you can automatically boot up an instance with a differently sized root volume. Flintrock does it.

In the case of this PR, I would suggest just setting the default root volume to a 30 GB SSD and refrain from adding new options. It's a lower friction change and should cover the use cases most people have, though I defer to @shivaram on whether he thinks that's a good idea.

[nchammas]

We should close this PR since spark-ec2 has been moved out of this repo and to a dedicated one.

Feel free to reopen this PR against the new repo. I suggest looking at how Flintrock resizes the root volumes, per my previous comment, before doing that though. I don't think we really need the new options.

[HyukjinKwon]

ping @kmaehashi