Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorporate Geekbench benchmark data into CDP #1

Open
vipulnaik opened this issue Jun 22, 2019 · 0 comments

Comments

Projects
None yet
2 participants
@vipulnaik
Copy link
Collaborator

commented Jun 22, 2019

I'm thinking of running Geekbench tests on all the EC2 instances and possibly the other cloud compute instances as well, and include the single-core score, multi-core score, and report link in CDP.

The reports look like this:

NOTE: https://aiimpacts.org/2015-flops-prices/ links to a Geekbench 3 report for c4.8xlarge from the section "Computing as a service". The link is from the word "delivers" in the para:

Amazon Elastic Cloud Compute (EC2) is a major seller of virtual computing. Based on their current pricing, renting a c4.8xlarge instance costs about $1.17 / hour.6 This is their largest instance optimized for computing performance (rather than e.g. memory). A c4.8xlarge instance delivers around 97.5 GFLOPS.7 This implies that a GFLOPShour costs $0.012. If we suppose this is an alternative to buying computer hardware, then the relevant time horizon is about three years. Over three years, renting this hardware will cost $316/GFLOPS, i.e. around two orders of magnitude more than buying GFLOPS in the form of GPUs.

Things to work out:

  • License: https://www.geekbench.com/versions/ To run it on EC2 instances we may need the Pro version (so that we have Command Line Tools). That has a one-time cost of $99. We should review the license terms carefully and make sure the product has the functionality we want.
  • What operating system and image to use for the tests: For instance, should we uniformly use Ubuntu 18 for all tests? Is it available across all the instance types that we are hoping to run the tests on?
  • Automation scripts to spin up the instance, run the tests, deposit the results safely, then exit the instance, and (either while running the instance or later) submit the report to Geekbench: We'll probably first make sure the scripts work well on the lowest-cost instances. We'll also use spot instances for EC2 to keep test-running costs low. Once the automation logic is figured out, we can run the tests on all the instances.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.