Spotsy - Amazon EC2 Spot Instance Analysis Framework

Spotsy is a Play! Framework web application for analyzing Amazon EC2 Spot prices. It has been developed independently, but is currently used at Kifi to reduce infrastructure costs. We have been able to cut our EC2 monthly bill by over 60% by using a spot-first environment.

Some EC2 Spot instances are chronically underpriced for a variety of reasons. Amazon, likely for good business reasons, doesn't make it simple to identify where signifiant discounts are available. Spotsy downloads spot price history, along with instance information, to easily compare which instances are underprices, relative to their CPU, memory, etc.

Currently, Spotsy is just the backend API — a featured frontend is currently under development.

This has also been a chance for me to experiment with Scala libraries that I've wanted to use. They include: cats for functional programming constructs, scalaz for Task, macwire for compile-time dependency injection, coursier as an Ivy replacement, and the latest version of Play. As such, a large part of this project is exploratory.

Example analysis

Spotsy efficiently stores and can generate statistical information about historical spot price information, in the context of instance size / ability. This allows you to make easy cost-per-compute-ability decisions. Here is small sample of what can be generated by Spotsy:

Name	Zone	ECU	Memory	$/week	99%ile $/hr	$/ECU	$/RAM	% Off
m2.xlarge	us-west-2a	6.5	17.1	2.784	0.017	0.002	0.001	92.9%
m2.xlarge	us-west-1a	6.5	17.1	2.783	0.019	0.003	0.001	92.8%
hi1.4xlarge	us-west-2b	35	60.5	27.78	0.226	0.006	0.003	92.6%
m2.xlarge	us-west-2b	6.5	17.1	2.861	0.018	0.002	0.001	92.4%
m2.xlarge	us-west-1b	6.5	17.1	2.933	0.021	0.003	0.001	92.1%
m1.large	us-west-1b	4	7.5	2.713	0.016	0.004	0.002	91.4%
m2.xlarge	us-west-2c	6.5	17.1	3.105	0.021	0.003	0.001	91.3%
r3.large	us-west-1a	6.5	15.25	2.738	0.016	0.002	0.001	91.1%
m1.xlarge	us-west-1a	8	15	5.486	0.033	0.004	0.002	91.0%
hi1.4xlarge	us-west-2c	35	60.5	27.8	0.285	0.008	0.004	90.7%
m1.large	us-west-1a	4	7.5	2.807	0.017	0.004	0.002	90.6%
m1.medium	us-west-2b	2	3.75	1.36	0.008	0.004	0.002	90.5%
m1.medium	us-west-2c	2	3.75	1.361	0.008	0.004	0.002	90.5%
m1.medium	us-west-2a	2	3.75	1.36	0.008	0.004	0.002	90.5%
m2.4xlarge	us-west-2b	26	68.4	13.24	0.092	0.003	0.001	90.5%
m1.large	us-west-2c	4	7.5	2.743	0.016	0.004	0.002	90.5%
m1.large	us-west-2b	4	7.5	2.759	0.016	0.004	0.002	90.5%
m1.large	us-west-2a	4	7.5	2.759	0.016	0.004	0.002	90.4%
r3.large	us-west-1b	6.5	15.25	2.862	0.018	0.002	0.001	90.2%
m1.xlarge	us-east-1b	8	15	5.553	0.034	0.004	0.002	90.2%
m2.4xlarge	us-west-2c	26	68.4	12.7	0.096	0.003	0.001	90.1%
i2.8xlarge	us-west-1a	104	244	126	0.75	0.007	0.003	90.0%

As you can see, significant discounts are available on the spot market. % Off is based on the regular demand price for that instance, compared to the previous week's realized spot price, given an infinite bid.

Running Spotsy

The only external dependency Spotsy has is sbt. Once installed, running sbt run will build and run the Play! web service on port 9000.

You will need to configure AWS keys by creating a file at conf/credentials. Here is an example stub:

aws {
  credentials {
    accessKey = "AAAAAAAA"
    secretKey = "BBBBBB+J/zhpX"
  }
}

play.crypto.secret = "010101010101"

Upon first request, if you have never run Spotsy before, spot price data will be downloaded and saved to disk. This may take quite some time due to AWS rate limits. If you're downloading for several zones and for a large time window (see HistoricalSpotPrice.scala to configure regions and startBackdate), the process may take several hours.