Helper scripts to launch disposable Disco clusters for map/reduce tasks using EC2 spot instances
- Ability to launch/reconfigure Disco clusters on the whim.
- Use cheaper AWS spot instances.
- Don't care about permanent storage in DDFS. Instead we use S3 or something else.
- Being able to (re)configure number of slaves and instance types.
- Using only Official Ubuntu Cloud Guest Amazon Machine Images (AMIs) - http://cloud.ubuntu.com/ami/
- Probably will use Instance store, since Map/Reduce is supposed to be more CPU bound than IO bound.
- Create config file with some defaults
$ python create_config.py > config.json
Edit config file based on Appendix below
Run this script many times to take care of pending tasks.
$ python create_cluster.py config.json
|BASE_PACKAGES||Yes||Do not override.|
|ADDITIONAL_PACKAGES||Yes||Additional packages to be installed using apt-get|
|PIP_REQUIREMENTS||Yes||Packages installed via pip. run after all apt-get tasks|
|AWS_ACCESS||Yes||REQUIRED. Your AWS access key|
|AWS_SECRET||Yes||REQUIRED. Your AWS secret key|
|AMI||Yes||What AMI to use. Hint: http://cloud.ubuntu.com/ami/|
|MAX_BID||Yes||How much to bid for each instance.|
|INSTANCE_TYPE||Yes||What instance type to use?|
|MASTER_INSTANCE_TYPE||No||Defaults to INSTANCE_TYPE|
|SLAVE_INSTANCE_TYPE||No||Defaults to INSTANCE_TYPE|
|KEY_NAME||Yes||The key name in EC2 console. To be able to login as ubuntu user for debugging|
|SECURITY_GROUPS||Yes||At-least 1 security group needed. Settings example next section.|
|MGMT_KEY||Yes||Public key(s) that is added on instances. Set this as your public keys. Multiple seperated by newline.|
|TAG_KEY||Yes||key used for tagging. If you are making multiple clusters, then make these ubique|
|NUM_SLAVES||Yes||Number of slaves|
|MASTER_MULTIPLIER||No||Ratio of workers compared to cores in master|
|SLAVE_MULTIPLIER||No||Ratio of workers compared to cores in each slave|
|POST_INIT||No||Bash scripts, runs as root after rest of the initialization|
- Allow all udp, tcp traffic from within the group on all ports.
- Allow ssh from 0.0.0.0/0 ... or atleast from your workstation.
- Ability to select region. Currently it is hardcoded to default (us-east)
- Specify different instance types with counts for slaves.
- Ability to stop unresponsive slaves.
- Ability to scale down cluster based on config changes. It can already scale up.
- Make pip installable, so that we can install it globally, and keep config file inside relavent project directories.