[google compute] add pricing data update script

[mbrukman]

Also used the script to update pricing.json with current prices.

/cc: @erjohnso, @Kami

[mbrukman]

Updated to handle difference in pricing on a per-region basis; please see the updated pricing.json for the new structure.

[erjohnso]

This looks good @mbrukman, thanks! @Kami, can you give this a quick look too?

Are you also planning on following up with a change to the GCE driver to pick up the pricing data? nudge, nudge, wink, wink

[Kami]

Ideally, we could pull out update_pricing_file function from https://github.com/apache/libcloud/blob/trunk/contrib/scrape-ec2-prices.py and re-use it in this script (we could move it to libcloud.pricing).

This way we can re-use this functionality and make sure keys are always sorted in the same order and format of the file stays the same.

[mbrukman]

@Kami — the update_pricing_file() function in that file is a complex way of saying json.dumps(indent=4, sort_keys=True) — it produces the same output. I used the Python script in this PR to regenerate the entire JSON file and did not affect anything around it other than adding the new "google" section.

My goal after committing this PR is to refactor the main() method in my script to become a general driver which reads the current JSON file, calls out to the different providers to update the structure, and then writes it out to disk. As you can see in my script, the phases are clearly delineated and the provider-specific update functions can be called from the middle of the script.

Thoughts?

[mbrukman]

@Kami — ping?

[davidwilson2038]

@mbrukman That is probably a good question for @erjohnso . Also, one other thing...I jumped the gun on my previous comments...just realized the other drivers use the name of the driver + region name for the name on the pricing json if they have different prices for regions. (e.g. ec2_ap_northeast). Having the region nested in the json as you have currently should break the _get_size_price method when called for list_sizes in the driver. To get this to work, we need to change the list_sizes method and have each region be its own entry in the json file (ie google_asia or google_europe, etc.)

[mbrukman]

I've updated this PR as follows:

• each region is a separate entry: google_asia, google_europe, google_us
• I added sorting to all keys, taking into account numeric substrings and treating them as numbers, so that an entry such as "12GB" sorts after "3GB" rather than before (lexicographically)
• reformatted the entire file so that it's now in the canonical format (which also converted numbers like 0.090 -> 0.09 possibly due to simplejson library)
• added a utility library to do the above and tests for it

[erjohnso]

Thanks for this @mbrukman! Sorry for the long wait.