Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[google compute] add pricing data update script #464

Closed
wants to merge 1 commit into from

Conversation

mbrukman
Copy link
Contributor

@mbrukman mbrukman commented Feb 17, 2015

Also used the script to update pricing.json with current prices.

/cc: @erjohnso, @Kami

@mbrukman
Copy link
Contributor Author

mbrukman commented Feb 17, 2015

Updated to handle difference in pricing on a per-region basis; please see the updated pricing.json for the new structure.

@mbrukman mbrukman force-pushed the google-cloud-pricing-json branch from 9ad9cb8 to da0ccd1 Compare Feb 17, 2015
@erjohnso
Copy link
Member

erjohnso commented Feb 17, 2015

This looks good @mbrukman, thanks! @Kami, can you give this a quick look too?

Are you also planning on following up with a change to the GCE driver to pick up the pricing data? nudge, nudge, wink, wink

@Kami
Copy link
Member

Kami commented Feb 18, 2015

Ideally, we could pull out update_pricing_file function from https://github.com/apache/libcloud/blob/trunk/contrib/scrape-ec2-prices.py and re-use it in this script (we could move it to libcloud.pricing).

This way we can re-use this functionality and make sure keys are always sorted in the same order and format of the file stays the same.

@mbrukman
Copy link
Contributor Author

mbrukman commented Feb 18, 2015

@Kami — the update_pricing_file() function in that file is a complex way of saying json.dumps(indent=4, sort_keys=True) — it produces the same output. I used the Python script in this PR to regenerate the entire JSON file and did not affect anything around it other than adding the new "google" section.

My goal after committing this PR is to refactor the main() method in my script to become a general driver which reads the current JSON file, calls out to the different providers to update the structure, and then writes it out to disk. As you can see in my script, the phases are clearly delineated and the provider-specific update functions can be called from the middle of the script.

Thoughts?

@mbrukman mbrukman force-pushed the google-cloud-pricing-json branch from da0ccd1 to 39be91b Compare Feb 22, 2015
@mbrukman mbrukman force-pushed the google-cloud-pricing-json branch 2 times, most recently from d11ad30 to d3e54ca Compare Sep 18, 2015
@mbrukman
Copy link
Contributor Author

mbrukman commented Sep 18, 2015

@Kami — ping?

@davidwilson2038
Copy link
Contributor

davidwilson2038 commented Oct 26, 2015

@mbrukman That is probably a good question for @erjohnso . Also, one other thing...I jumped the gun on my previous comments...just realized the other drivers use the name of the driver + region name for the name on the pricing json if they have different prices for regions. (e.g. ec2_ap_northeast). Having the region nested in the json as you have currently should break the _get_size_price method when called for list_sizes in the driver. To get this to work, we need to change the list_sizes method and have each region be its own entry in the json file (ie google_asia or google_europe, etc.)

@mbrukman mbrukman force-pushed the google-cloud-pricing-json branch from d3e54ca to 82b77b3 Compare Nov 4, 2015
Also used the script to update  `pricing.json` with current prices on a
per-region basis. Added numeric sorting to the JSON output; as a result, all
other pricing is now sorted numerically, e.g., "12GB" sorts after "3GB" and
"150" sorts after "20" whereas previously, it would be the opposite.
@mbrukman mbrukman force-pushed the google-cloud-pricing-json branch from 82b77b3 to 3ecb3ea Compare Nov 4, 2015
@mbrukman
Copy link
Contributor Author

mbrukman commented Nov 4, 2015

I've updated this PR as follows:

  • each region is a separate entry: google_asia, google_europe, google_us
  • I added sorting to all keys, taking into account numeric substrings and treating them as numbers, so that an entry such as "12GB" sorts after "3GB" rather than before (lexicographically)
  • reformatted the entire file so that it's now in the canonical format (which also converted numbers like 0.090 -> 0.09 possibly due to simplejson library)
  • added a utility library to do the above and tests for it

@asfgit asfgit closed this in c07dbe8 Nov 9, 2015
@erjohnso
Copy link
Member

erjohnso commented Nov 9, 2015

Thanks for this @mbrukman! Sorry for the long wait.

@mbrukman mbrukman deleted the google-cloud-pricing-json branch Nov 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants