Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
AndreasSko committed Nov 12, 2019
1 parent ffcee40 commit c3c8441
Showing 1 changed file with 75 additions and 7 deletions.
82 changes: 75 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Galaxy Benchmarker
A tool for benchmarking Galaxy job-destinations.
A tool for benchmarking Galaxy job-destinations.

The goal is to easily benchmark different job-destinations. All you have to do is to configure the benchmarker itself,
define the destinations you want to benchmark, the workflows you want to run and the benchmarks you want to use. GalaxyBenchmarker
Expand All @@ -15,7 +15,7 @@ be run per destination and if some ansible-playbooks should be run before or aft
What is the difference between running a workflow for the first time or for it having been run multiple
times already? What is the overhead for staging time, installing tools, etc?

This benchmark cleans up Pulsar for every cold run, using a Ansible-Playbook ([coldwarm_pretask.yml]).
This benchmark cleans up Pulsar for every cold run, using a Ansible-Playbook ([coldwarm_pretask.yml]).

#### Destination Comparison
What are the differences between multiple destinations in terms of staging time (sending data
Expand All @@ -33,9 +33,77 @@ How does a destination handle a big burst of requests?
* Ansible

## Usage
* Configure the benchmarker. Examples can be found in [benchmark_config.yml.example].
* Run
### Install all dependencies
Some additional packages are needed in order for GalaxyBenchmarker to function properly.
To install those, run the following command:
```shell
pip3 install -r requirements.txt
```

python galaxy_benchmarker

* Analyze metrics in InfluxDB. Example Dashboards for Grafana can be found at [./grafana_dashboards]
### Configure the GalaxyBenchmarker
To use the benchmarker, you first need to create a yaml-configuration file.
We need to have access to a Galaxy instance in order to submit workflows:
```yaml
galaxy:
url: https://usegalaxy.eu
user_key: YOUR-KEY
```

Next, we need to define the workflows that should be run. The benchmarker
uses the test functionality of [Planemo](https://github.com/galaxyproject/planemo) to
submit workflows. Examples can be found at: https://github.com/usegalaxy-eu/workflow-testing. For a start, you
can clone this repository and use those workflows for benchmarking. Workflows are defined in the
configuration as following:
```yaml
workflows:
- name: ARDWorkflow
type: Galaxy
path: /../workflow-testing/sklearn/ard/ard.ga
```

The benchmarker can submit jobs to multiple job destinations to compare the performance of each. For routing
the jobs to the right destination, users can be defined for each. The GalaxyBenchmarker can take
care of that
(see the [configuration examples](https://github.com/AndreasSko/Galaxy-Benchmarker/blob/master/benchmark_config.yml.example))
if you are an administrator for the Galaxy instance. Otherwise, it is also possible to use different
users for each destination and link them in the configuration of the benchmarker:
```yaml
destinations:
- name: Destination1
type: Galaxy
galaxy_user_name: USERNAME
galaxy_user_key: USERKEY
```

Now, we just need to define the actual benchmark that we want to perform. To compare different job destinations,
you can do the following:
```yaml
benchmarks:
- name: DestinationComparisonBenchmark
type: DestinationComparison
destinations:
- Destination1
workflows:
- ARDWorkflow
runs_per_workflow: 1
```

### Run the benchmarks
The GalaxyBenchmarker can use different configuration files. If none is given, it will look for `benchmark_config.yml`
```shell
python3 galaxy_benchmarker --config benchmark_config.yml
```

## Additional options
### InfluxDB
In normal cases, GalaxyBenchmarker will save the results in a json file under the `results` directory. However, jobs
metrics can also be submitted to InfluxDB for further analysis.
```yaml
influxdb:
host: influxdb.example.com
port: 8086
username: glx_benchmarker_user
password: supersecret
db_name: glx_benchmarker
```
Example Dashboards for Grafana can be found at [grafana_dashboards](https://github.com/AndreasSko/Galaxy-Benchmarker/tree/master/grafana_dashboards)

0 comments on commit c3c8441

Please sign in to comment.