New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[google compute] add pricing data update script #464

Closed
wants to merge 1 commit into
base: trunk
from

Conversation

Projects
None yet
4 participants
@mbrukman
Contributor

mbrukman commented Feb 17, 2015

Also used the script to update pricing.json with current prices.

/cc: @erjohnso, @Kami

@mbrukman

This comment has been minimized.

Show comment
Hide comment
@mbrukman

mbrukman Feb 17, 2015

Contributor

Updated to handle difference in pricing on a per-region basis; please see the updated pricing.json for the new structure.

Contributor

mbrukman commented Feb 17, 2015

Updated to handle difference in pricing on a per-region basis; please see the updated pricing.json for the new structure.

@erjohnso

This comment has been minimized.

Show comment
Hide comment
@erjohnso

erjohnso Feb 17, 2015

Member

This looks good @mbrukman, thanks! @Kami, can you give this a quick look too?

Are you also planning on following up with a change to the GCE driver to pick up the pricing data? nudge, nudge, wink, wink

Member

erjohnso commented Feb 17, 2015

This looks good @mbrukman, thanks! @Kami, can you give this a quick look too?

Are you also planning on following up with a change to the GCE driver to pick up the pricing data? nudge, nudge, wink, wink

@Kami

This comment has been minimized.

Show comment
Hide comment
@Kami

Kami Feb 18, 2015

Member

Ideally, we could pull out update_pricing_file function from https://github.com/apache/libcloud/blob/trunk/contrib/scrape-ec2-prices.py and re-use it in this script (we could move it to libcloud.pricing).

This way we can re-use this functionality and make sure keys are always sorted in the same order and format of the file stays the same.

Member

Kami commented Feb 18, 2015

Ideally, we could pull out update_pricing_file function from https://github.com/apache/libcloud/blob/trunk/contrib/scrape-ec2-prices.py and re-use it in this script (we could move it to libcloud.pricing).

This way we can re-use this functionality and make sure keys are always sorted in the same order and format of the file stays the same.

@mbrukman

This comment has been minimized.

Show comment
Hide comment
@mbrukman

mbrukman Feb 18, 2015

Contributor

@Kami — the update_pricing_file() function in that file is a complex way of saying json.dumps(indent=4, sort_keys=True) — it produces the same output. I used the Python script in this PR to regenerate the entire JSON file and did not affect anything around it other than adding the new "google" section.

My goal after committing this PR is to refactor the main() method in my script to become a general driver which reads the current JSON file, calls out to the different providers to update the structure, and then writes it out to disk. As you can see in my script, the phases are clearly delineated and the provider-specific update functions can be called from the middle of the script.

Thoughts?

Contributor

mbrukman commented Feb 18, 2015

@Kami — the update_pricing_file() function in that file is a complex way of saying json.dumps(indent=4, sort_keys=True) — it produces the same output. I used the Python script in this PR to regenerate the entire JSON file and did not affect anything around it other than adding the new "google" section.

My goal after committing this PR is to refactor the main() method in my script to become a general driver which reads the current JSON file, calls out to the different providers to update the structure, and then writes it out to disk. As you can see in my script, the phases are clearly delineated and the provider-specific update functions can be called from the middle of the script.

Thoughts?

@mbrukman

This comment has been minimized.

Show comment
Hide comment
@mbrukman

mbrukman Sep 18, 2015

Contributor

@Kami — ping?

Contributor

mbrukman commented Sep 18, 2015

@Kami — ping?

@davidwilson2038

This comment has been minimized.

Show comment
Hide comment
@davidwilson2038

davidwilson2038 Oct 26, 2015

Contributor

@mbrukman That is probably a good question for @erjohnso . Also, one other thing...I jumped the gun on my previous comments...just realized the other drivers use the name of the driver + region name for the name on the pricing json if they have different prices for regions. (e.g. ec2_ap_northeast). Having the region nested in the json as you have currently should break the _get_size_price method when called for list_sizes in the driver. To get this to work, we need to change the list_sizes method and have each region be its own entry in the json file (ie google_asia or google_europe, etc.)

Contributor

davidwilson2038 commented Oct 26, 2015

@mbrukman That is probably a good question for @erjohnso . Also, one other thing...I jumped the gun on my previous comments...just realized the other drivers use the name of the driver + region name for the name on the pricing json if they have different prices for regions. (e.g. ec2_ap_northeast). Having the region nested in the json as you have currently should break the _get_size_price method when called for list_sizes in the driver. To get this to work, we need to change the list_sizes method and have each region be its own entry in the json file (ie google_asia or google_europe, etc.)

[google compute] add pricing data update script
Also used the script to update  `pricing.json` with current prices on a
per-region basis. Added numeric sorting to the JSON output; as a result, all
other pricing is now sorted numerically, e.g., "12GB" sorts after "3GB" and
"150" sorts after "20" whereas previously, it would be the opposite.
@mbrukman

This comment has been minimized.

Show comment
Hide comment
@mbrukman

mbrukman Nov 4, 2015

Contributor

I've updated this PR as follows:

  • each region is a separate entry: google_asia, google_europe, google_us
  • I added sorting to all keys, taking into account numeric substrings and treating them as numbers, so that an entry such as "12GB" sorts after "3GB" rather than before (lexicographically)
  • reformatted the entire file so that it's now in the canonical format (which also converted numbers like 0.090 -> 0.09 possibly due to simplejson library)
  • added a utility library to do the above and tests for it
Contributor

mbrukman commented Nov 4, 2015

I've updated this PR as follows:

  • each region is a separate entry: google_asia, google_europe, google_us
  • I added sorting to all keys, taking into account numeric substrings and treating them as numbers, so that an entry such as "12GB" sorts after "3GB" rather than before (lexicographically)
  • reformatted the entire file so that it's now in the canonical format (which also converted numbers like 0.090 -> 0.09 possibly due to simplejson library)
  • added a utility library to do the above and tests for it

@asfgit asfgit closed this in c07dbe8 Nov 9, 2015

@erjohnso

This comment has been minimized.

Show comment
Hide comment
@erjohnso

erjohnso Nov 9, 2015

Member

Thanks for this @mbrukman! Sorry for the long wait.

Member

erjohnso commented Nov 9, 2015

Thanks for this @mbrukman! Sorry for the long wait.

@mbrukman mbrukman deleted the mbrukman:google-cloud-pricing-json branch Nov 10, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment