Skip to content
Permalink
Browse files
Expanding and clarifying readme files for using aws and scripts. (#102)
  • Loading branch information
DonalEvans authored and pivotal-jbarrett committed Aug 26, 2019
1 parent 8e29d0b commit 439b85762ed5d8261930c82bfc449cd9ab1ef41d
Showing 2 changed files with 59 additions and 20 deletions.
@@ -13,19 +13,17 @@ test configuration and test execution framework.
These benchmarks are under development.

## Running the benchmarks
The benchmarks require machines with passwordless ssh enabled in order to run.
So ensure that the authentication key pair for SSH does not have a passphrase. If you had
already previously created a key pair with a passphrase, you can create a different key pair which
is of a different type than the previously created one. Public key needs to be in PEM format. Some newer OpenSSH
versions default to a new format. Use `-m PEM` to force PEM format.
The benchmarks require machines with passwordless ssh enabled in order to run, so ensure that the authentication key pair for SSH does not have a passphrase. If you have previously created a key pair with a passphrase, you can create a different key pair which
is of a different type than the previously created one. Be sure to backup your existing key pair before creating a new one. The public key needs to be in PEM format, but some newer OpenSSH
versions default to a new format. PEM format can be forced by using `-m PEM`:
```
ssh-keygen -m PEM -t rsa
```
While runinng a test on a single machine (i.e. localhost) add the generated key to `authorized_keys` to authorize the user.
While runinng a test on a single machine (i.e. localhost) add the generated key to `authorized_keys` to authorize the user:
```
cat <your_public_key_file> >> ~/.ssh/authorized_keys
```
Test if you can ssh to localhost.
Test if you can ssh to localhost:
```
ssh localhost
```
@@ -59,9 +57,9 @@ Options:
-d : Debug
-i : Info
```
### Running in aws
### Scripts for running in aws and analyzing results

This project includes some scripts to automate running benchmarks in AWS. See the
This project includes some scripts to automate running benchmarks in AWS and analyzing the results produced (as well as the results produced from running locally). See the
[README.md](infrastructure/scripts/aws/README.md) in the infrastructure/aws directory.

## Project structure
@@ -72,12 +70,12 @@ The project is divided into two modules

## Sample benchmark

Benchmarks are defined in a declarative configuration classes. Each configuration class is run as
Benchmarks are defined in declarative configuration classes. Each configuration class is run as
a junit test which calls the configure method and passes it to the TestRunner, which executes
the test on the provided TEST_HOSTS.

Benchmarks are composed of `before tasks`, `after tasks`, and `workload tasks`. The `before` and `after`
tasks are run once. `Workload` tasks are run repeatedly and their execution time is measured and
Benchmarks are composed of `before` tasks, `after` tasks, and `workload` tasks. Each seperate `before` and `after`
task is run once. `Workload` tasks are run repeatedly and their execution time is measured and
reported by the yardstick framework.

```java
@@ -4,11 +4,23 @@ These utilities create instances and run tests in your AWS account

# Prerequisites
* You must have the aws cli installed. If `aws` is not on your path then you can try to install it with `pip3 install awscli --upgrade --user`. See [Amazon's aws cli installation instructions](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
* You must also set your secret key for the CLI. You must set up a profile named `geode-benchmarks`, so use the command `aws configure --profile geode-benchmarks` to configure the CLI. You will need to specify the "AWS Access Key ID" and "AWS Secret Access Key". You can get these from an existing team member. Set the "Default region name" to "us-west-2". See [Amazon's instructions](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)
* You must also set your secret key for the CLI. You must set up a profile named `geode-benchmarks`, so use the command `aws configure --profile geode-benchmarks` to configure the CLI. You will need to specify the "AWS Access Key ID" and "AWS Secret Access Key," which can be obtained from a team member. Set the "Default region name" to "us-west-2". See [Amazon's instructions](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html)

Alternately, if you already have AWS credentials and just need to create the `geode-benchmarks` profile, you can first modify your existing AWS credentials file (found at `~/.aws/credentials`) and add the following lines:

[geode-benchmarks]
aws_access_key_id = Your access ID key
aws_secret_access_key = Your secret access key

Then modify the `config` file found in the same directory and add the following:

[profile geode-benchmarks]
region = us-west-2


# Image

If you need to build the image, you must have packer installed. But you can run the following scripts (launch, run, destroy) without building the image.
If you need to build the image, you must have packer installed. The following scripts (launch, run, analyze, destroy) can be run without building the image.
Build the image in the image directory using the `build_image.sh` script.


@@ -20,13 +32,13 @@ One of the ways that can be done is via environment variables.
prompt> aws configure
AWS Access Key ID [****************foo1]:
AWS Secret Access Key [****************bar2]:
Default region name [us-west-1]:
Default region name [us-west-2]:

Export environment variables as follows.

export AWS_ACCESS_KEY_ID=myaccesskeyfoo1
export AWS_SECRET_ACCESS_KEY=mysecretaccesskeybar2
export AWS_REGION="us-west-1"
export AWS_REGION="us-west-2"


# launch_cluster.sh
@@ -85,12 +97,29 @@ Options:
-B|--gbb|--baseline-branch|--baseline-geode-branch : Geode Baseline Branch (default: develop)
-m|--metadata : Test metadata to output to file, comma-delimited (optional)
--ci : Set when the instances are being started for use in Continuous Integration
-- : All subsequent arguments are passed to the benchmark tast as arguments
-- : All subsequent arguments are passed to the benchmark task as arguments
-h|-?|--help : Help message

e.g. ./run_against_baseline.sh -t test_environment -v <sha1 of target version> -V <sha1 of base version> -R <baseline repo e.g. user/geode> -B <baseline branch name> -b <target branch name> -r <target repo e.g. user/geode>


# analyze_tests.sh
Compares the results of two benchmark runs and outputs analysis of their relative performance.

Usage:

./analyze_tests.sh [-o <output directory> | [--baselineDir <baseline directory> --branchDir <branch directory>]] [options ...] [-- arguments ...]

Options:

-o|--output|--outputDir : The directory containing benchmark results
--baseline|--baselineDir : The directory containing baseline benchmark results
--branch|--branchDir : The directory containing branch benchmark results
--ci : Set if starting instances for Continuous Integration
-- : All subsequent arguments are passed to the benchmark task as arguments.
-h|--help : This help message


# destroy_cluster.sh
Destroys a cluster that you created.

@@ -105,12 +134,24 @@ Options:
-h|-?|--help : Help message


#Example
## Example

Example 1 - Generating and comparing two benchmark runs using run_tests.sh and analyze_tests.sh.

The first command creates a new cluster with 4 instances and the tag "mycluster" using launch_cluster.sh.

The second command runs only the `MyCustomBenchmark` benchmark test (by using the `-- --tests=MyCustomBenchmark` argument) found on the `myBenchmarkBranch` branch of the `myGit/geode-benchmarks` repository. This benchmark runs against the develop branch of Geode, adds some metadata, and outputs the results to `~/benchmarking/baseline` using run_test.sh.

The third command runs the same benchmark against the `myGeodeBranch` branch of the `myGit/geode` repository and outputs the results to `~/benchmarking/branch` using run_tests.sh.

The fourth command compares the results found in `~/benchmarking/branch` to the results found in `~/benchmarking/baseline` and outputs analysis of the operations per second and latency of the branch benchmark relative to the baseline using analyze_tests.sh.

Example 1 - run_test.sh:
The fifth command destroys the cluster using destroy_cluster.sh.
```bash
./launch_cluster.sh --tag mycluster --count 4
./run_tests.sh --tag mycluster --geode-branch develop --benchmark-branch benchmarkBranch --metadata "'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'"
./run_tests.sh --tag mycluster --geode-branch develop --benchmark-repo myGit/geode-benchmarks --benchmark-branch myBenchmarkBranch --metadata "'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" --output ~/benchmarking/baseline -- --tests=MyCustomBenchmark
./run_tests.sh --tag mycluster --geode-repo myGit/geode --geode-branch myGeodeBranch --benchmark-repo myGit/geode-benchmarks --benchmark-branch myBenchmarkBranch --metadata "'name':'HelenaTestingCPUs','CPU':'256','geodeBranch':'CPUTest'" --output ~/benchmarking/branch -- --tests=MyCustomBenchmark
./analyze_tests.sh --branch ~/benchmarking/branch --baseline ~/benchmarking/baseline
./destroy_cluster.sh --tag mycluster
```

0 comments on commit 439b857

Please sign in to comment.