Automated deployment of instances on EC2 via plugin for high CPU/GPU applications at the lowest price.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Dockerfile
README.md
cloudstomp.py

README.md

Cloudstomp - Automated deployment of instances on EC2 via plugin for high CPU/GPU applications at the lowest price

Cloudstomp is a python application that is in active development and built to make deployment of high CPU/GPU applications in Amazon EC2 as spot instances for affordable, quick, and easy data analysis.

Plugins can be created in the json format that enable users to quickly deploy applications. Two plugins are included for demonstration, one for a hello world application, and one for hashcat.

Install

Installation can be completed with Docker or natively. For a native local installation, start by downloading cloudstomp:

# cd /usr/bin
# wget https://raw.githubusercontent.com/Fmstrat/cloudstomp/master/cloudstomp.py
# chmod +x cloudstomp.py

or, if you would like to use Docker:

# alias cloudstomp.py='docker run -it --rm --name=cloudstomp -v ${HOME}/.cloudstomp:/data cloudstomp /cloudstomp.py'

Plugins

The current documentation for plugins and avaialable plugins can be found at https://github.com/Fmstrat/cloudstomp-plugins and are:

  • Hello World - The example shown below
  • Hashcat - Password cracking in the cloud

AWS Credentials

To use Cloudstomp you must provide AWS credentials. To set up your credentials at the AWS Management Console:

  • Sign in at https://console.aws.amazon.com/iam/
  • Go to Users
  • Then Add User
  • Add the user cloudstomp
  • Check the box for Programmatic access
  • Click Next: Permissions
  • If the Administrators group doesn't exist, click Add group
  • Enter Administrators for Group name
  • Check Administrator Access
  • Click Create group which will return you to the add user screen
  • Back on the add user screen, choose Add user to group and check Administrators
  • Click Next: Tags
  • Click Next: Review
  • Click Create user
  • Access key ID and Secret access key are displayed which are required for configuration of Cloudstomp

Usage

First, we need to download the latest plugins:

# cloudstomp.py --update

This will download all of the preconfigured plugins into the ~/cloudstomp folder (or the folder specified in the Docker command).

Then, we fire up cloudstomp:

# cloudstomp.py

The first time Cloudstomp runs, it requires the AWS credentials. This sets up a credentials file completely seperate from those used in your normal AWS administration to reduce conflicts.

AWS CLI is not configured, would you like to do this now? [y/N]: y
Access key, Secret key, and default region must all be configured.
AWS Access Key ID [None]: *********
AWS Secret Access Key [None]: *********
Default region name [None]: us-east-1
Default output format [None]: 

Then, follow the prompts:

Sessions:
-------------------------------------------
 c) Create
 q) Quit
-------------------------------------------
Select session: c

Creating a new session.
Plugins:
-------------------------------------------
 1) Hashcat
 2) Hello World
-------------------------------------------
Please select a plugin: 2

The first time you choose a plugin, the AMI that instances will be created from needs to be built. This is a one-time item and the AMI is stored on S3 as they always are for faster future deployment.

The AMI for hello world needs to be built.
Would you like to build it now? [Y/n]: y

[2019-01-18 10:57:47] Build will take place on a t2.nano instance.
[2019-01-18 10:57:47] Checking availability zones for t2.nano...
[2019-01-18 10:57:48] Getting subnet...
[2019-01-18 10:57:49]   Using subnet subnet-0bea6a005f7f8dbed.
[2019-01-18 10:57:49] Checking VPC...
[2019-01-18 10:57:50]   Using VPC vpc-02d8172ca01a496de.
[2019-01-18 10:57:50] Checking security group...
[2019-01-18 10:57:50]   Creating new security group.
[2019-01-18 10:57:51]   Adding rules...
[2019-01-18 10:57:52]     Added port 22.
[2019-01-18 10:57:52]   Using security group sg-019fedf5084f3e068.
[2019-01-18 10:57:54] Instance i-06f82a99547d8d66e started.
[2019-01-18 10:57:54] Waiting for boot to complete...
[2019-01-18 10:57:54]   Instance in pending state, sleeping 15s...
[2019-01-18 10:58:10] Instance has public IP of 18.212.124.241.
[2019-01-18 10:58:10] Checking for SSH connectivity...
[2019-01-18 10:58:13]   Waiting for SSH to start, sleeping 15s...
[2019-01-18 10:58:28]   Waiting for SSH to start, sleeping 15s...
[2019-01-18 10:58:46] Starting build scripts...
[2019-01-18 10:58:46]   Running build.sh...
[2019-01-18 10:58:48]     Uploaded.
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [83.2 kB]
Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]

... [ Output of setting up the instance (apt-get/etc) ] ...

Successfully built PyYAML
Installing collected packages: docutils, jmespath, six, python-dateutil, urllib3, botocore, PyYAML, pyasn1, rsa, colorama, futures, s3transfer, awscli
Successfully installed PyYAML-3.13 awscli-1.16.91 botocore-1.12.81 colorama-0.3.9 docutils-0.14 futures-3.2.0 jmespath-0.9.3 pyasn1-0.4.5 python-dateutil-2.7.5 rsa-3.4.2 s3transfer-0.1.13 six-1.12.0 urllib3-1.24.1
[2019-01-18 10:59:59] Stopping instance...
[2019-01-18 11:00:00]   Instance in stopping state, sleeping 15s...
[2019-01-18 11:00:16]   Instance in stopping state, sleeping 15s...
[2019-01-18 11:00:37] Creating AMI image...
[2019-01-18 11:00:38]   Build is in pending state, sleeping 15s...
[2019-01-18 11:00:54]   Build is in pending state, sleeping 15s...
[2019-01-18 11:01:09]   Build is in pending state, sleeping 15s...
[2019-01-18 11:01:25]   Build is in pending state, sleeping 15s...
[2019-01-18 11:01:41]   Build is in pending state, sleeping 15s...
[2019-01-18 11:01:57]   Build is in pending state, sleeping 15s...
[2019-01-18 11:02:13]   Build is in pending state, sleeping 15s...
[2019-01-18 11:02:33] Tagging with build version...
[2019-01-18 11:02:34] Terminating source image...
[2019-01-18 11:02:35] Build complete.

Returning to session creation.

Once the build completes, we return to creating the new session. You will notice one of the questions below is about encryption. Output data is stored on S3 for later download when processes finish. This data can be encrypted with gocryptfs, that way if it is stored for a long period of time it is more secure. For the Hello World plugin, data is syncronized to S3 every 15 seconds. For other plugins, the reocmmendation is every 300 seconds.

Would you like to run the instance On-Demand or as a Spot Instance? [o/S]: s
Instances:
-------------------------------------------
 1) 1 vCPU, 1GB RAM     [t2.micro]
 2) 1 vCPU, 2GB RAM     [t2.small]
-------------------------------------------
Please select an instance: 1

Pricing:
--------------  ----------      ----------      ----------      ------------
Zone            Average         Latest          On Demand       Reccomended
--------------  ----------      ----------      ----------      ------------
1) us-east-1c   $0.003500       $0.003500       $0.011600       $0.004025 (best)
2) us-east-1f   $0.003500       $0.003500       $0.011600       $0.004025
3) us-east-1b   $0.003500       $0.003500       $0.011600       $0.004025
4) us-east-1a   $0.003500       $0.003500       $0.011600       $0.004025
5) us-east-1e   $0.003500       $0.003500       $0.011600       $0.004025
6) us-east-1d   $0.003500       $0.003500       $0.011600       $0.004025
--------------  ----------      ----------      ----------      ------------
(The default is 15% over the average past two week spot price to keep your instance active.)

Which zone would you like to run in? 1
What is the max price you would pay for a spot instance? [0.004025]: 
What would you like to call this instance? Testing
How many seconds would you like to wait between printing 'hello world'? [10]: 10
Enter a password for gocryptfs encrypted output (leave blank for no encryption): 
Session Testing created.

Sessions:
-------------------------------------------
 1) Testing
 c) Create
 q) Quit
-------------------------------------------
Select session: 

Now you have the session Testing available. Next, we will start up an instance running the Hello World plugin.

Sessions:
-------------------------------------------
 1) Testing
 c) Create
 q) Quit
-------------------------------------------
Select session: 1

Testing:
-------------------------------------------
 1) Start
 2) Output
 s) Show variables
 d) Delete
 b) Back
-------------------------------------------
Choose an option: 1

[2019-01-18 11:06:27] Getting subnet...
[2019-01-18 11:06:28]   Using subnet subnet-0bea6a005f7f8dbed.
[2019-01-18 11:06:28] Checking VPC...
[2019-01-18 11:06:29]   Using VPC vpc-02d8172ca01a496de.
[2019-01-18 11:06:29] Checking security group...
[2019-01-18 11:06:30]   Found existing security group.
[2019-01-18 11:06:30]   Using security group sg-019fedf5084f3e068.
[2019-01-18 11:06:31] Waiting on spot id sir-9jcg74mg...
[2019-01-18 11:06:31]   Spot request in open state, sleeping 15s...
[2019-01-18 11:07:03]   Instance in running state.
[2019-01-18 11:07:04] Instance has public IP of 54.88.86.34.
[2019-01-18 11:07:04] Checking for SSH connectivity...
[2019-01-18 11:07:07]   Waiting for SSH to start, sleeping 15s...
[2019-01-18 11:07:25] Uploading cloudstomp AWS credentials...
[2019-01-18 11:07:26] Uploading any supporting files...
[2019-01-18 11:07:26]   Complete.
[2019-01-18 11:07:26] Starting init script...
[2019-01-18 11:07:26]   Running remote.sh...
[2019-01-18 11:07:27]     Uploaded.
[2019-01-18 11:07:33] Launch complete.

Testing:
-------------------------------------------
 1) Status
 2) Connect
 3) Stop
 4) Output
 s) Show variables
 d) Delete
 b) Back
-------------------------------------------
Choose an option: 

Now we can check on the status of our instance, connect to it and attach to the screen session, stop the instance, or show the outputs from S3. Outputs are available even after the instance is stopped.

Testing:
-------------------------------------------
 1) Status
 2) Connect
 3) Stop
 4) Output
 s) Show variables
 d) Delete
 b) Back
-------------------------------------------
Choose an option: 1
[2019-01-18 11:09:59] Instance has public IP of 54.88.86.34.
[2019-01-18 11:09:59] Checking for SSH connectivity...
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Testing:
-------------------------------------------
 1) Status
 2) Connect
 3) Stop
 4) Output
 s) Show variables
 d) Delete
 b) Back
-------------------------------------------
Choose an option: 3
[2019-01-18 11:10:28] Instance has public IP of 54.88.86.34.
[2019-01-18 11:10:28] Checking for SSH connectivity...
There is no screen to be resumed matching helloworld.
There is no screen to be resumed matching sync.
upload: cloudstomp/plain/output.txt to s3://cloudstomp-testing/output.txt
upload: cloudstomp/plain/sync.txt to s3://cloudstomp-testing/sync.txt
Connection to 54.88.86.34 closed.

Sessions:
-------------------------------------------
 1) Testing     [shutting-down]
 c) Create
 q) Quit
-------------------------------------------
Select session: 1

Testing:
-------------------------------------------
 1) Output
 s) Show variables
 d) Delete
 b) Back
-------------------------------------------
Choose an option: 1

download: s3://cloudstomp-testing/output.txt to ../../../tmp/cloudstomp-testing/plain/output.txt
download: s3://cloudstomp-testing/sync.txt to ../../../tmp/cloudstomp-testing/plain/sync.txt
Starting hello world...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
Hello world!
Sleeping 10 seconds...
^C
-------------------------------------------

Testing:
-------------------------------------------
 1) Output
 s) Show variables
 d) Delete
 b) Back
-------------------------------------------
Choose an option: 

Many of these options can also be executed from the command line (check with cloudstomp.py --help) for automation.