Amazon EC2 Spot Instance Analysis Framework
Scala HTML JavaScript
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app
conf
project
public
test
.gitignore
LICENSE
README.md
build.sbt
scalastyle-config.xml

README.md

Spotsy - Amazon EC2 Spot Instance Analysis Framework

Spotsy is a Play! Framework web application for analyzing Amazon EC2 Spot prices. It has been developed independently, but is currently used at Kifi to reduce infrastructure costs. We have been able to cut our EC2 monthly bill by over 60% by using a spot-first environment.

Some EC2 Spot instances are chronically underpriced for a variety of reasons. Amazon, likely for good business reasons, doesn't make it simple to identify where signifiant discounts are available. Spotsy downloads spot price history, along with instance information, to easily compare which instances are underprices, relative to their CPU, memory, etc.

Currently, Spotsy is just the backend API — a featured frontend is currently under development.

This has also been a chance for me to experiment with Scala libraries that I've wanted to use. They include: cats for functional programming constructs, scalaz for Task, macwire for compile-time dependency injection, coursier as an Ivy replacement, and the latest version of Play. As such, a large part of this project is exploratory.

Example analysis

Spotsy efficiently stores and can generate statistical information about historical spot price information, in the context of instance size / ability. This allows you to make easy cost-per-compute-ability decisions. Here is small sample of what can be generated by Spotsy:

Name Zone ECU Memory $/week 99%ile $/hr $/ECU $/RAM % Off
m2.xlarge us-west-2a 6.5 17.1 2.784 0.017 0.002 0.001 92.9%
m2.xlarge us-west-1a 6.5 17.1 2.783 0.019 0.003 0.001 92.8%
hi1.4xlarge us-west-2b 35 60.5 27.78 0.226 0.006 0.003 92.6%
m2.xlarge us-west-2b 6.5 17.1 2.861 0.018 0.002 0.001 92.4%
m2.xlarge us-west-1b 6.5 17.1 2.933 0.021 0.003 0.001 92.1%
m1.large us-west-1b 4 7.5 2.713 0.016 0.004 0.002 91.4%
m2.xlarge us-west-2c 6.5 17.1 3.105 0.021 0.003 0.001 91.3%
r3.large us-west-1a 6.5 15.25 2.738 0.016 0.002 0.001 91.1%
m1.xlarge us-west-1a 8 15 5.486 0.033 0.004 0.002 91.0%
hi1.4xlarge us-west-2c 35 60.5 27.8 0.285 0.008 0.004 90.7%
m1.large us-west-1a 4 7.5 2.807 0.017 0.004 0.002 90.6%
m1.medium us-west-2b 2 3.75 1.36 0.008 0.004 0.002 90.5%
m1.medium us-west-2c 2 3.75 1.361 0.008 0.004 0.002 90.5%
m1.medium us-west-2a 2 3.75 1.36 0.008 0.004 0.002 90.5%
m2.4xlarge us-west-2b 26 68.4 13.24 0.092 0.003 0.001 90.5%
m1.large us-west-2c 4 7.5 2.743 0.016 0.004 0.002 90.5%
m1.large us-west-2b 4 7.5 2.759 0.016 0.004 0.002 90.5%
m1.large us-west-2a 4 7.5 2.759 0.016 0.004 0.002 90.4%
r3.large us-west-1b 6.5 15.25 2.862 0.018 0.002 0.001 90.2%
m1.xlarge us-east-1b 8 15 5.553 0.034 0.004 0.002 90.2%
m2.4xlarge us-west-2c 26 68.4 12.7 0.096 0.003 0.001 90.1%
i2.8xlarge us-west-1a 104 244 126 0.75 0.007 0.003 90.0%

As you can see, significant discounts are available on the spot market. % Off is based on the regular demand price for that instance, compared to the previous week's realized spot price, given an infinite bid.

Running Spotsy

The only external dependency Spotsy has is sbt. Once installed, running sbt run will build and run the Play! web service on port 9000.

You will need to configure AWS keys by creating a file at conf/credentials. Here is an example stub:

aws {
  credentials {
    accessKey = "AAAAAAAA"
    secretKey = "BBBBBB+J/zhpX"
  }
}

play.crypto.secret = "010101010101"

Upon first request, if you have never run Spotsy before, spot price data will be downloaded and saved to disk. This may take quite some time due to AWS rate limits. If you're downloading for several zones and for a large time window (see HistoricalSpotPrice.scala to configure regions and startBackdate), the process may take several hours.