EC2 and pricing updates [v2] by pmhahn · Pull Request #649 · apache/libcloud

pmhahn · 2015-12-02T10:40:39Z

The EC2 Instance Type information contains some errors (especially the disk size for the m3 instances), conversion errors between GiB an M[i]B, disk count, ..
Also g2.8xlarge and t2.large are missing.

The values on <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls> are given in [GiB]. Let Python convert them to [MiB], as the current values are a mix of [MB] and [MiB].

The values on <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls> are given in [GB]. Let Python calculate the size of multiple disks. Remove wrong factor 1000 from m3 instances.

Add vCPU information from <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls>

From <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls>

i2 are I/O optimized, while d2 are dense optimized

t2.micro is also a "Burstable Performance General Purpose" instance type.

Add source URLs. Add unit type for RAM and disk size.

Commit b7d851f added the pricing and instance info multiple times.

jimbobhickville · 2015-12-02T14:20:47Z

I think you missed a trailing comma here.

Thanks, added and force-pushed

jimbobhickville · 2015-12-02T14:25:16Z

Wow, that must have been super tedious. I'm surprised Amazon doesn't provide an API for this information. It looked good to me apart from that missing comma, but I don't have a lot of context around AWS so someone else more familiar should probably review.

contrib/scrape-ec2-prices.py changes the ordering of data/pricing.json, as it sorts the dictionary keys by string, which puts '10' before '2'. Add a function to split the dictionary keys into the numeric, alphabetic and other part and compare the first as integer, the middle by meaning, and the later as string.

Please note that the scraper stores the price as string and does not beautify the string to have exactly 3 digits in the radix part.

Remove duplicate description for NodeState.STOPPED added by commit b09c1be

pmhahn · 2015-12-02T15:03:47Z

I only slightly improved the sorting of EC2 scraper written mainly by Tomaz Muraus, which basically used the data from https://a0.awsstatic.com/pricing/1/ec2/linux-od.min.js. That could also be used to fill in the data in <libcloud/compute/drivers/ec2.py>, but that's for another day.
But yes, finding those mistakes of mixing 1000 and 1024 was some work ;-)

tonybaloney · 2015-12-03T04:25:40Z

@pmhahn Why has outscale been completely removed?

@tonybaloney No it hasn't been removed conmpletely, only the duplicate entries: it's a JSON dictionary, where the key is unique and only the last(?) entry wins:
Warning: the following script will create the files [1-8].py in your current directory!

curl -s https://raw.githubusercontent.com/apache/libcloud/trunk/libcloud/compute/drivers/ec2.py | sed -ne ' 1567,1593w 1.py 1594,1620w 2.py 1621,1647w 3.py 1648,1674w 4.py 1729,1755w 5.py 1756,1782w 6.py 1783,1809w 7.py 1810,1836w 8.py ' diff 1.py 2.py ; diff 2.py 3.py ; diff 3.py 4.py diff 5.py 6.py ; diff 6.py 7.py ; diff 7.py 8.py

Yep, eu-west-2 exists up above in the elided part of the diff, line 1293.

tonybaloney · 2015-12-03T04:26:51Z

@pmhahn that basically undoes #645 I think you made an error in merging (resyncing from trunk).

Kami · 2015-12-03T05:52:38Z

Yeah, there is contrib/scrape-ec2-prices.py, but it's possible that it's not working anymore since some of those .js files have been deprecated for a while now 😞

pmhahn · 2015-12-03T08:11:27Z

@tonybaloney: My reading is, that #645 is wrong, since OUTSCALE_(SAS|INC)_REGION_DETAILS is a dictionary and the keys "eu-west-2", "us-east-2" are used multiple times:

$ curl -s https://raw.githubusercontent.com/apache/libcloud/trunk/libcloud/data/pricing.json | sed -rne '/^ {8}"/p' | sort | uniq -d
        "osc_inc_eu_west_2": {
        "osc_inc_us_east_2": {
        "osc_sas_eu_west_2": {
        "osc_sas_us_east_2": {
$ curl -s https://raw.githubusercontent.com/apache/libcloud/trunk/libcloud/compute/drivers/ec2.py | sed -rne "/^OUTSCALE_SAS_REGION_DETAILS/,/^}/{/^ {4}'/p}" | sort | uniq -d
    'eu-west-2': {
    'us-east-2': {
$ curl -s https://raw.githubusercontent.com/apache/libcloud/trunk/libcloud/compute/drivers/ec2.py | sed -rne "/^OUTSCALE_INC_REGION_DETAILS/,/^}/{/^ {4}'/p}" | sort | uniq -d
    'eu-west-2': {
    'us-east-2': {

@Kami: You wan't me not not update the EC2 pricing? At least adding the prices for g2.8xlarge and t2.large is required to not break the unit test.

tonybaloney · 2015-12-04T05:22:43Z

@pmhahn yep good point, i see the mistake now.

Kami · 2015-12-04T09:26:49Z

LGTM

tonybaloney · 2015-12-04T20:13:10Z

Ok @pmhahn I'll close and merge this. FYI @LordShion see above.

pmhahn added 8 commits December 2, 2015 14:53

EC2: Specify RAM size in GiB

f3aed3f

The values on <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls> are given in [GiB]. Let Python convert them to [MiB], as the current values are a mix of [MB] and [MiB].

EC2: Fix disk size

d9d97bc

The values on <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls> are given in [GB]. Let Python calculate the size of multiple disks. Remove wrong factor 1000 from m3 instances.

EC2: Add missing vCPU information

7507735

Add vCPU information from <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls>

EC2: Add g2.8xlarge and t2.large

c68e0b1

From <https://aws.amazon.com/ec2/instance-types/?nc1=f_ls>

EC2: Rename i2 and d2 instance types

0d4490d

i2 are I/O optimized, while d2 are dense optimized

EC2: Move comment before t2 instances

0fc18e0

t2.micro is also a "Burstable Performance General Purpose" instance type.

EC2: Add clarifications

022f3cb

Add source URLs. Add unit type for RAM and disk size.

OSC: Remove duplicate entries

6c292c3

Commit b7d851f added the pricing and instance info multiple times.

pmhahn force-pushed the trunk branch from 3291f70 to 65a22a1 Compare December 2, 2015 13:53

jimbobhickville reviewed Dec 2, 2015
View reviewed changes

pmhahn added 3 commits December 2, 2015 15:57

EC2: Update pricing

85ece6a

Please note that the scraper stores the price as string and does not beautify the string to have exactly 3 digits in the radix part.

compute: Remove duplicate state STOPPED

f7ee6bc

Remove duplicate description for NodeState.STOPPED added by commit b09c1be

pmhahn force-pushed the trunk branch from 65a22a1 to f7ee6bc Compare December 2, 2015 14:58

tonybaloney reviewed Dec 3, 2015
View reviewed changes

asfgit closed this in 2f51455 Dec 4, 2015

Conversation

pmhahn commented Dec 2, 2015

Uh oh!

jimbobhickville Dec 2, 2015

Choose a reason for hiding this comment

Uh oh!

pmhahn Dec 2, 2015

Choose a reason for hiding this comment

Uh oh!

jimbobhickville commented Dec 2, 2015

Uh oh!

pmhahn commented Dec 2, 2015

Uh oh!

tonybaloney Dec 3, 2015

Choose a reason for hiding this comment

Uh oh!

pmhahn Dec 3, 2015

Choose a reason for hiding this comment

Uh oh!

jimbobhickville Dec 3, 2015

Choose a reason for hiding this comment

Uh oh!

tonybaloney commented Dec 3, 2015

Uh oh!

Kami commented Dec 3, 2015

Uh oh!

pmhahn commented Dec 3, 2015

Uh oh!

tonybaloney commented Dec 4, 2015

Uh oh!

Kami commented Dec 4, 2015

Uh oh!

tonybaloney commented Dec 4, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants