Skip to content

Commit

Permalink
feat(qa): take parameters from command arguments in python scripts (#…
Browse files Browse the repository at this point in the history
…2665)

The main changes are to allow setting the scripts' parameters by passing
arguments instead of have them hardcoded in variables. The rest is just
refactoring to give some structure to the code.


---

#### PR checklist

- [ ] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use
[unclog](https://github.com/informalsystems/unclog) to manage our
changelog)
- [X] Updated relevant documentation (`docs/` or `spec/`) and code
comments
- [X] Title follows the [Conventional
Commits](https://www.conventionalcommits.org/en/v1.0.0/) spec

---------

Co-authored-by: lasaro <lasaro@informal.systems>
  • Loading branch information
hvanz and lasarojc committed Apr 4, 2024
1 parent dc4590e commit 839d76e
Show file tree
Hide file tree
Showing 4 changed files with 352 additions and 352 deletions.
78 changes: 30 additions & 48 deletions scripts/qa/reporting/README.md
@@ -1,6 +1,6 @@
# Reporting Scripts

This directory contains some utility scripts used in the reporting/QA.
This directory contains some utility scripts used for generating reports of QA processes.

* [`latency_throughput.py`](./latency_throughput.py) is a Python script that uses
[matplotlib] to plot a graph of transaction latency vs throughput rate based on
Expand All @@ -17,8 +17,8 @@ This directory contains some utility scripts used in the reporting/QA.

## Setup

Execute the following within this directory (the same directory as the
`latency_throughput.py` file).
Before running the Python scripts, execute the following within this directory (the same directory
as the `latency_throughput.py` file).

```bash
# Create a virtual environment into which to install your dependencies
Expand All @@ -32,66 +32,48 @@ pip install -r requirements.txt
```

## Latency vs Throughput Plotting
To show the instructions and parameter options, execute

To show the instructions and parameter options, execute
```bash
./latency_throughput.py --help
```
Be sure that the virtual environment is enabled before running the script.

Example:

For example, the following command will generate a PNG file called `cmt_v1.png` in the current
directory based on the `raw.csv` file generated by the reporting tool. The `-t` flag overrides the
default title at the top of the plot.
```bash
# Do the following while ensuring that the virtual environment is activated (see
# the Setup steps).
#
# This will generate a plot in a PNG file called 'tm034.png' in the current
# directory based on the reporting tool CSV output in the "raw.csv" file. The
# '-t' flag overrides the default title at the top of the plot.

./latency_throughput.py \
-t 'CometBFT v0.34.x Latency vs Throughput' \
./tm034.png \
/path/to/csv/files/raw.csv
./latency_throughput.py -t 'CometBFT v1.x Latency vs Throughput' ./cmt_v1.png /path/to/results/raw.csv
```

## Latency vs Throughput Plotting (version 2)
Example:

The `latency_plotter.py` script generates a series of plots in the `imgs` folder.
Plots include combined experiment plots and experiments as subplots.
- `all_experiments`: plots of all experiments as individual subplots.
- `all_configs`: plots of all experiments, grouped by configuration (r,c).
- `cXrY.png`: Independent plot of experiments of configuration (c=X,r=Y) as different curves.
- `cXrY_merged.png`: Independent plot of experiments of configuration (c=X,r=Y) combined as single curve.
- `e_ID.png`: independent plot with just experiment with id ID as a single curve.

Example:
```bash
# Do the following while ensuring that the virtual environment is activated (see
# the Setup steps).
#
# This will generate a series of plots in the `imgs` folder.
# Plots include combined experiment plots and experiments as subplots.
# - all_experiments - plots of all experiments as individual subplots.
# - all_configs - plots of all experiments, grouped by configuration (r,c).
# cXrY.png - Independent plot of experiments of configuration (c=X,r=Y) as different curves.
# cXrY_merged.png - Independent plot of experiments of configuration (c=X,r=Y) combined as single curve.
# e_ID.png - independent plot with just experiment with id ID as a single curve.

mkdir -p imgs
python3 latency_plotter.py /path/to/csv/files/raw.csv
./latency_plotter.py v1.0.0-alpha.2 /path/to/results/raw.csv
```
Be sure that the virtual environment is enabled before running the script.

## Prometheus metrics

1. Ensure that Prometheus is running locally and listening on port 9090.
2. Tweak the script to your needs
1. Adjust the time window
2. Select the right fork
3. Select the right test case
4. Tweak/add/remove metrics
3. Run the script as follows
```bash
# Do the following while ensuring that the virtual environment is activated (see
# the Setup steps).
#
# This will generate a series of plots in the folder `imgs` of the current folder.

mkdir imgs
python3 prometheus_plotter.py
```
4. Plots are saved in the `imgs` folder.
The `prometheus_plotter.py` script generates a series of plots in the folder `imgs` of the current folder.

Before running the script, check that a Prometheus server in `localhost:9090`. This is the default URL hardcoded in the script.

Run the script from the virtual environment as follows:
```bash
./prometheus_plotter.py <release_name> <start_time> <window_size> <test_case>
```

For details and examples of how to run the script, just run `python3 prometheus_plotter.py`

[matplotlib]: https://matplotlib.org/
[pandas]: https://pandas.pydata.org

0 comments on commit 839d76e

Please sign in to comment.