Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[google compute] add pricing data update script #464

Closed
wants to merge 1 commit into from

Conversation

@mbrukman
Copy link
Contributor

mbrukman commented Feb 17, 2015

Also used the script to update pricing.json with current prices.

/cc: @erjohnso, @Kami

mbrukman referenced this pull request Feb 17, 2015
@mbrukman mbrukman force-pushed the mbrukman:google-cloud-pricing-json branch from eb51181 to 9ad9cb8 Feb 17, 2015
@mbrukman
Copy link
Contributor Author

mbrukman commented Feb 17, 2015

Updated to handle difference in pricing on a per-region basis; please see the updated pricing.json for the new structure.

@mbrukman mbrukman force-pushed the mbrukman:google-cloud-pricing-json branch from 9ad9cb8 to da0ccd1 Feb 17, 2015
@erjohnso
Copy link
Member

erjohnso commented Feb 17, 2015

This looks good @mbrukman, thanks! @Kami, can you give this a quick look too?

Are you also planning on following up with a change to the GCE driver to pick up the pricing data? nudge, nudge, wink, wink

@Kami
Copy link
Member

Kami commented Feb 18, 2015

Ideally, we could pull out update_pricing_file function from https://github.com/apache/libcloud/blob/trunk/contrib/scrape-ec2-prices.py and re-use it in this script (we could move it to libcloud.pricing).

This way we can re-use this functionality and make sure keys are always sorted in the same order and format of the file stays the same.

@mbrukman
Copy link
Contributor Author

mbrukman commented Feb 18, 2015

@Kami — the update_pricing_file() function in that file is a complex way of saying json.dumps(indent=4, sort_keys=True) — it produces the same output. I used the Python script in this PR to regenerate the entire JSON file and did not affect anything around it other than adding the new "google" section.

My goal after committing this PR is to refactor the main() method in my script to become a general driver which reads the current JSON file, calls out to the different providers to update the structure, and then writes it out to disk. As you can see in my script, the phases are clearly delineated and the provider-specific update functions can be called from the middle of the script.

Thoughts?

@mbrukman mbrukman force-pushed the mbrukman:google-cloud-pricing-json branch from da0ccd1 to 39be91b Feb 22, 2015
@mbrukman mbrukman force-pushed the mbrukman:google-cloud-pricing-json branch 2 times, most recently from d11ad30 to d3e54ca Sep 18, 2015
@mbrukman
Copy link
Contributor Author

mbrukman commented Sep 18, 2015

@Kami — ping?

@davidwilson2038
Copy link
Contributor

davidwilson2038 commented Oct 26, 2015

@mbrukman That is probably a good question for @erjohnso . Also, one other thing...I jumped the gun on my previous comments...just realized the other drivers use the name of the driver + region name for the name on the pricing json if they have different prices for regions. (e.g. ec2_ap_northeast). Having the region nested in the json as you have currently should break the _get_size_price method when called for list_sizes in the driver. To get this to work, we need to change the list_sizes method and have each region be its own entry in the json file (ie google_asia or google_europe, etc.)

@mbrukman mbrukman force-pushed the mbrukman:google-cloud-pricing-json branch from d3e54ca to 82b77b3 Nov 4, 2015
Also used the script to update  `pricing.json` with current prices on a
per-region basis. Added numeric sorting to the JSON output; as a result, all
other pricing is now sorted numerically, e.g., "12GB" sorts after "3GB" and
"150" sorts after "20" whereas previously, it would be the opposite.
@mbrukman mbrukman force-pushed the mbrukman:google-cloud-pricing-json branch from 82b77b3 to 3ecb3ea Nov 4, 2015
@mbrukman
Copy link
Contributor Author

mbrukman commented Nov 4, 2015

I've updated this PR as follows:

  • each region is a separate entry: google_asia, google_europe, google_us
  • I added sorting to all keys, taking into account numeric substrings and treating them as numbers, so that an entry such as "12GB" sorts after "3GB" rather than before (lexicographically)
  • reformatted the entire file so that it's now in the canonical format (which also converted numbers like 0.090 -> 0.09 possibly due to simplejson library)
  • added a utility library to do the above and tests for it
@asfgit asfgit closed this in c07dbe8 Nov 9, 2015
@erjohnso
Copy link
Member

erjohnso commented Nov 9, 2015

Thanks for this @mbrukman! Sorry for the long wait.

@mbrukman mbrukman deleted the mbrukman:google-cloud-pricing-json branch Nov 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.