Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,15 @@ tools_software_languages:
operatingsystems:
- Linux


further_reading:
- resource:
title: Effective Go
link: https://go.dev/doc/effective_go#performance
type: blog
- resource:
title: Benchmark testing in Go
link: https://dev.to/stefanalfbo/benchmark-testing-in-go-17dc
type: blog

### FIXED, DO NOT MODIFY
# ================================================================================
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ In this Learning Path, you'll compare performance using two four-core GCP instan

{{% notice Note %}}
Arm-based c4a-standard-4 instances and Intel-based c4-standard-8 instances both utilize four cores. Both instances are categorized by GCP as members a series that demonstrates consistent high performance.
The main difference between the two is that c4a has 16 GB of RAM, while c4 has 30 GB of RAM. This Learning Path uses equivalent core counts to ensure a fair performance comparison.
The main difference between the two is that c4a has 16 GB of RAM, while c4 has 30 GB of RAM. This Learning Path uses equivalent core counts as an example of performance comparison.
{{% /notice %}}


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,66 +6,65 @@ weight: 53
layout: learningpathall
---

In the last section, you learned how to run benchmarks and Benchstat manually. Now you'll automate that process and generate visual reports using a script called `rexec_sweet.py`.
In the last section, you learned how to run benchmarks and Benchstat manually. Now you'll automate that process and generate visual reports using a tools called `rexec_sweet`.

## What is rexec_sweet.py?
## What is rexec_sweet?

`rexec_sweet.py` is a script that automates the benchmarking workflow: it connects to your GCP instances, runs benchmarks, collects results, and generates HTML reports - all in one step.
`rexec_sweet` is a Python project available on GitHub that automates the benchmarking workflow. It connects to your GCP instances, runs benchmarks, collects results, and generates HTML reports - all in one step.

It provides several key benefits:

- **Automation**: Runs benchmarks on multiple VMs without manual SSH connections
- **Consistency**: Ensures benchmarks are executed with identical parameters
- **Visualization**: Generates HTML reports with interactive charts for easier analysis

Before running the script, ensure you've completed the "Install Go, Sweet, and Benchstat" step. All other dependencies are installed automatically by the setup script.
Before running the tool, ensure you've completed the "Install Go, Sweet, and Benchstat" step. All other dependencies are installed automatically by the installer.

## Set up rexec_sweet

Follow the steps below to set up rexec_sweet.py.
Follow the steps below to set up `rexec_sweet`.

### Create a working directory

On your local machine, open a terminal, then create and change into a directory to store the `rexec_sweet.py` script and related files:
On your local machine, open a terminal, and create a new directory:

```bash
mkdir rexec_sweet
cd rexec_sweet
```
```bash
mkdir rexec_sweet
cd rexec_sweet
```

### Clone the repository

Get the `rexec_sweet.py` script from the GitHub repository:
Get `rexec_sweet` from GitHub:

```bash
git clone https://github.com/geremyCohen/go_benchmarks.git
cd go_benchmarks
```
```bash
git clone https://github.com/geremyCohen/go_benchmarks.git
cd go_benchmarks
```

### Run the installer

Copy and paste this command into your terminal to run the installer:

```bash
./install.sh
```
```bash
./install.sh
```

If the install.sh script detects that you already have dependencies installed, it might ask you if you want to reinstall them:
If the installer detects that you already have dependencies installed, it might ask you if you want to reinstall them:

```output
pyenv: /Users/gercoh01/.pyenv/versions/3.9.22 already exists
continue with installation? (y/N)
```
```output
pyenv: /Users/gercoh01/.pyenv/versions/3.9.22 already exists
continue with installation? (y/N)
```

If you see this prompt, enter `N` to continue with the installation without modifying the existing installed dependencies.
If you see this prompt, enter `N` to continue with the installation without modifying the existing installed dependencies.

### Verify VM status

Make sure the GCP VM instances you created in the previous section are running. If not, start them now, and wait a few minutes for them to finish booting.

{{% notice Note %}}
The install script prompts you to authenticate with Google Cloud Platform (GCP) using the gcloud command-line tool at the end of install. If after installing you have issues running the script and/or get GCP authentication errors, you can manually authenticate with GCP by running the following command: `gcloud auth login`
The installer prompts you to authenticate with Google Cloud Platform (GCP) using the gcloud command-line tool at the end of install. If after installing you have issues running or you get GCP authentication errors, you can manually authenticate with GCP by running the following command: `gcloud auth login`
{{% /notice %}}


Continue on to the next section to run the script and see how it simplifies the benchmarking process.
Continue on to the next section to run the tool and see how it simplifies the benchmarking process.
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,18 @@ weight: 54
### FIXED, DO NOT MODIFY
layout: learningpathall
---

With `rexec_sweet` installed, your benchmarking instances running, and your local machine authenticated with GCP, you're ready to run automated benchmarks across your configured environments.

## Run an automated benchmark and generate results

To begin, open a terminal on your local machine and run:

```bash
rexec_sweet
rexec-sweet
```
The script will prompt you to choose a benchmark.

The tool will prompt you to choose a benchmark.

Press **Enter** to run the default benchmark, markdown, which is a good starting point for your first run.

Expand All @@ -33,7 +35,7 @@ Available benchmarks:
Enter number (1-10) [default: markdown]:
```

The script then detects your running GCP instances and displays them. You’ll be asked whether you want to use the first two instances it finds and the default install paths.
The tool then detects your running GCP instances and displays them. You’ll be asked whether you want to use the first two instances it finds and the default install paths.

```output
Available instances:
Expand Down Expand Up @@ -70,7 +72,7 @@ Output directory: /private/tmp/a/go_benchmarks/results/c4-c4a-markdown-20250610T
...
```

After selecting instances and paths, the script will:
After selecting instances and paths, the tool will:
- Run the selected benchmark on both VMs
- Use `benchstat` to compare the results
- Push the results to your local machine
Expand All @@ -89,13 +91,13 @@ Report generated in results/c4-c4a-markdown-20250610T190407

Once on your local machine, `rexec_sweet` will generate an HTML report that opens automatically in your web browser.

If you close the report, you can reopen it by navigating to the `results` subdirectory and opening report.html in your browser.
If you close the report, you can reopen it by navigating to the `results` subdirectory and opening report.html in your browser.

![alt-text#center](images/run_auto/2.png "Sample HTML report")


{{% notice Note %}}
If you see output messages from `rexec_sweet.py` similar to "geomeans may not be comparable" or "Dn: ratios must be >0 to compute geomean", this is expected and can be ignored. These warnings typically appear when benchmark sets differ slightly between the two VMs.
If you see output messages similar to "geomeans may not be comparable" or "Dn: ratios must be >0 to compute geomean", this is expected and can be ignored. These warnings typically appear when benchmark sets differ slightly between the two VMs.
{{% /notice %}}

Upon completion, the script generates a report in the `results` subdirectory of the current working directory of the `rexec_sweet.py` script, which opens automatically in your web browser to view the benchmark results and comparisons.
Upon completion, the tool generates a report in the `results` subdirectory of the current working directory, which opens automatically in your web browser to view the benchmark results and comparisons.
Original file line number Diff line number Diff line change
Expand Up @@ -98,15 +98,15 @@ Large gaps between average and peak memory usage suggest opportunities for memor

Here are some general tips to keep in mind as you explore benchmarking across different apps and instance types:

- Unlike Intel and AMD processors that use hyper-threading, Arm processors provide single-threaded cores without hyper-threading. A four-core Arm processor has four independent cores running four threads, while a four-core Intel processor provides eight logical cores through hyper-threading. This means that each Arm vCPU represents a full physical core, while each Intel/AMD vCPU represents half a physical core. For fair comparison, this Learning Path uses a 4-vCPU Arm instance against an 8-vCPU Intel instance. When scaling up instance sizes during benchmarking, make sure to keep a 2:1 Intel/AMD:Arm vCPU ratio if you wish to keep parity on CPU resources.
- On Intel and AMD processors with hyper-threading, each vCPU corresponds to a logical core (hardware thread), and two vCPUs share a single physical core. On Arm processors (which do not use hyper-threading), each vCPU corresponds to a full physical core. For comparison, this Learning Path uses a 4-vCPU Arm instance against an 8-vCPU Intel instance, maintaining a 2:1 Intel:Arm vCPU ratio to keep parity on physical CPU resources.

- Run each benchmark at least 10 times (-count 10) to account for outliers and produce statistically meaningful results.
- Run each benchmark at least 10 times to account for outliers and produce statistically meaningful results.

- Results can be bound by CPU, memory, or I/O performance. If you see significant differences in one metric but not others, it might indicate a bottleneck in that area; running the same benchmark with different configurations (for example, using more CPU cores or more memory) can help identify the bottleneck.

- Results can be bound by CPU, memory, or I/O performance. If you see significant differences in one metric but not others, it might indicate a bottleneck in that area; running the same benchmark with different configurations (for example, using more CPU cores or more memory) can help identify the bottleneck.