Ec2 build with spot instance pricing

[ck37]

Here are the json changes needed to build with spot instance pricing (https://www.packer.io/docs/builders/amazon-ebs.html#spot_price):

Add to the amazon-ebs builder in BCE-summer-2015.json:
"spot_price": "auto",
"spot_price_auto_product": "Linux/UNIX",

(Actual patch to come sometime.)

From a small sample size (n=3, k=1), this drops the m3.medium instance pricing from $0.06-$0.07 to $0.001 (83% cost reduction) and only took an extra few minutes to get the spot instance, even at noon.

We could then use the cost-savings to use a beefier instance than m3.medium and potentially reduce the build time. I'm not sure how sensitive the build time is to the instance power though.

I'm also guessing that since the build time is not that long, we won't need to worry about the instance getting terminated before the build completes. I don't know how realistic that is.

[paciorek]

I think I'm missing something - how long does it take to build? 0.07 /
0.0031 is large but 0.07 - 0.0031 is pennies...

On Thu, Jul 16, 2015 at 1:09 PM, Chris Kennedy notifications@github.com
wrote:

Here are the json changes needed to build with spot instance pricing (
https://www.packer.io/docs/builders/amazon-ebs.html#spot_price):

Add to the amazon-ebs builder in BCE-summer-2015.json:
"spot_price": "auto",
"spot_price_auto_product": "Linux/UNIX",

(Actual patch to come sometime.)

From a small sample size (n=3, k=1), this drops the m3.medium instance
pricing from $0.06-$0.07 to $0.0031 (95% cost reduction) and only took an
extra few minutes to get the spot instance, even at noon.

We could then use the cost-savings to use a beefier instance than
m3.medium and potentially reduce the build time. I'm not sure how sensitive
the build time is to the instance power though.

I'm also guessing that since the build time is not that long, we won't
need to worry about the instance getting terminated before the build
completes. I don't know how realistic that is.

—
Reply to this email directly or view it on GitHub
#54.

[ck37]

The absolute magnitude is not too huge, true.

My thought is that the time to build could be improved to aid development iteration - right now it's 1-2 hours for me, using the m3.medium default. So if we switch to spot pricing and then use some more powerful instance + configuration, perhaps we could get a faster build time without adding to the cost.

Perhaps the build time could be reduced through extra cores, more ram, more IOPS, or higher clock speed. Looking at https://www.packer.io/docs/builders/amazon-ebs.html#volume_type we could specify a high-IOPS build disk for example.

[paciorek]

If I were doing it and paying out of our SCF funds (which is what is
happening now), I would just use a more powerful instance and not worry
about the very modest cost increase. So I'd prefer to leave non-spot as
the default, but you of course are welcome to build using spot if you like.

On Thu, Jul 16, 2015 at 2:12 PM, Chris Kennedy notifications@github.com
wrote:

The absolute magnitude is not too huge, true.

My thought is that the time to build could be improved to aid development
iteration - right now it's 1-2 hours for me, using the m3.medium default.
So if we switch to spot pricing and then use some more powerful instance +
configuration, perhaps we could get a faster build time without adding to
the cost.

Perhaps the build time could be reduced through extra cores, more ram,
more IOPS, or higher clock speed. Looking at
https://www.packer.io/docs/builders/amazon-ebs.html#volume_type we could
specify a high-IOPS build disk for example.

—
Reply to this email directly or view it on GitHub
#54 (comment).

[ck37]

Okie dokie, that sounds completely fair. I'll close this and just do it manually as desired. I'll keep investigating potential instance configurations to reduce build times though.