diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md index 820701deea..4b2e191e5e 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/_index.md @@ -1,22 +1,18 @@ --- -title: Go Benchmarks with Sweet and Benchstat - -draft: true -cascade: - draft: true +title: Benchmark Go performance with Sweet and Benchstat minutes_to_complete: 60 -who_is_this_for: This is an introductory topic for developers who are interested in measuring the performance of Go-based applications on Arm-based servers. +who_is_this_for: This introductory topic is for developers who want to measure and compare the performance of Go applications on Arm-based servers. -learning_objectives: - - Learn how to start up Arm64 and x64 instances of GCP VMs - - Install Go, benchmarks, benchstat, and sweet on the two VMs - - Use sweet and benchstat to compare the performance of Go applications on the two VMs +learning_objectives: + - Provision Arm64 and x86_64 VM instances on Google Cloud + - Install Go, Sweet, and Benchstat on each VM instance + - Run benchmarks and use Benchstat to compare Go application performance across architectures prerequisites: - - A [Google Cloud account](https://console.cloud.google.com/). This learning path can be run on on-prem or on any cloud provider instance, but specifically documents the process for running on Google Axion. - - A local machine with [Google Cloud CLI](/install-guides/gcloud/) installed. + - A [Google Cloud account](https://console.cloud.google.com/). This Learning Path can be run on any cloud provider or on-premises, but it focuses on Google Cloud’s Axion Arm64-based instances. + - A local machine with [Google Cloud CLI](/install-guides/gcloud/) installed author: Geremy Cohen diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md index 70b37ecf2e..f7b9ad3a57 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4_vm.md @@ -1,32 +1,31 @@ --- -title: Launching a Intel Emerald Rapids Instance +title: Launch an Intel Emerald Rapids c4-standard-8 instance weight: 30 ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Section Overview -In this section, you will set up the second benchmarking system, an Intel Emerald Rapids `c4-standard-8` instance. +In this section, you'll set up the second benchmarking system: an Intel-based Emerald Rapids `c4-standard-8` instance on Google Cloud (referred to as **c4**). -## Creating the Instance +## Create the c4-standard-8 instance -To create the second system, follow the previous lesson's c4a install instructions, but make the following changes: +Follow the same steps from the previous section where you launched the c4a instance, but make the following changes for the Intel-based c4-standard-8: -1. **Name your instance:** For the `Name` field, enter "c4". +* In the **Name** field, enter "c4". +* In the **Machine types for common workloads** section, select the **c4** radio button. +![alt-text#center](images/launch_c4/3.png "Select the c4 radio button") -2. **Select machine series:** Scroll down to the Machine series section, and select the C4 radio button. +* In the **Machine configuration** section, open the dropdown select `c4-standard-8`. -![](images/launch_c4/3.png) +![alt-text#center](images/launch_c4/4.png "Open the dropdown and select `c4-standard-8`") -3. **View machine types:** Scroll down to the Machine type dropdown, and click it to show all available options. +* In the **Machine type** section, open the dropdown and select `c4-standard-8` under the **Standard** tab. -![](images/launch_c4/4.png) +![alt-text#center](images/launch_c4/5.png "Select `c4-standard-8`") -4. **Choose machine size:** Select "c4-standard-8" under the Standard tab. - -![](images/launch_c4/5.png) - -{{% notice Note %}} Don't forget to set the disk size for this c4 to 1000GB under the "OS and Storage" tab like you did for the c4a.{{% /notice %}} +{{% notice Note %}} +Be sure to set the disk size to **1000 GB** in the **OS and Storage** tab, just as you did for the `c4a` instance. +{{% /notice %}} After the c4 instance starts up, you are ready to continue to the next section, where you'll install the benchmarking software. diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md index 106352dc7c..e05e7222e9 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/add_c4a_vm.md @@ -1,67 +1,62 @@ --- -title: Launching a Google Axion Instance +title: Launch an Arm-based c4a-standard-4 instance weight: 20 ### FIXED, DO NOT MODIFY layout: learningpathall --- -## Overview -In this section, you'll learn how to spin up the first of two different VMs used for benchmarking Go tests, an Arm-based Google Axion c4a-standard-4 (c4a for short). +In this section, you'll launch the first of two VMs used for benchmarking Go applications: the Arm-based c4a-standard-4 instance on Google Cloud, (referred to as "c4a"). -## Creating the c4a-standard-4 Instance +## Create the c4a-standard-4 instance -1. **Access Google Cloud Console:** Navigate to [https://console.cloud.google.com/welcome](https://console.cloud.google.com/welcome) +Go to the Google Cloud console: [https://console.cloud.google.com/welcome](https://console.cloud.google.com/welcome). -2. **Search for VM instances:** Click into the Search field. +In the search bar at the top, start typing `vm`, then select **VM instances** when it appears. -3. **Find VM Instances:** Start typing `vm` until the UI auto-completes `VM Instances`, then click it. +![alt-text#center](images/launch_c4a/3.png "Select VM instances") -![](images/launch_c4a/3.png) + On the **VM instances** page, click **Create instance**. -The VM Instances page appears. +![alt-text#center](images/launch_c4a/4.png "Select Create instance") -4. **Create a new instance:** Click `Create instance` + In the **Name** field, enter the name of the instance - here it should be `c4a`. -![](images/launch_c4a/4.png) +![alt-text#center](images/launch_c4a/5.png "Enter name of the instance") -The Machine configuration page appears. +Now select the machine series by scrolling down to the Machine series section, and selecting the **C4A** radio button. -5. **Name your instance:** Click the `Name` field, and enter "c4a" for the `Name`. +![alt-text#center](images/launch_c4a/7.png "Select C4A radio button") -![](images/launch_c4a/5.png) +To view machine types, scroll down to the **Machine type** dropdown, and select it to show all available options. -6. **Select machine series:** Scroll down to the Machine series section, and select the C4A radio button. +![alt-text#center](images/launch_c4a/8.png "Select Machine type dropdown") -![](images/launch_c4a/7.png) +Now choose machine size by selecting **c4a-standard-4** under the **Standard** tab. -7. **View machine types:** Scroll down to the Machine type dropdown, and click it to show all available options. +![alt-text#center](images/launch_c4a/9.png "Select machine size") -![](images/launch_c4a/8.png) +To configure storage, select the **OS and Storage** tab. -8. **Choose machine size:** Select "c4a-standard-4" under the Standard tab. +![alt-text#center](images/launch_c4a/10.png "Configure storage") -![](images/launch_c4a/9.png) +To modify storage settings, select **Change**. -9. **Configure storage:** Click the "OS and Storage" tab. +![alt-text#center](images/launch_c4a/11.png "Modify storage settings") -![](images/launch_c4a/10.png) +To set disk size, select the **Size (GB)** field and enter "1000" for the value. -10. **Modify storage settings:** Click "Change" +![alt-text#center](images/launch_c4a/16.png "Enter value in the Size (GB) field") -![](images/launch_c4a/11.png) +Now confirm storage settings by selecting **Select** to continue. -11. **Set disk size:** Double-click the "Size (GB)" field, then enter "1000" for the value. +![alt-text#center](images/launch_c4a/18.png "Confirm the selection of settings with the Select button") -![](images/launch_c4a/16.png) +To launch the instance, select **Create** to bring up the instance. -12. **Confirm storage settings:** Click "Select" to continue. +![alt-text#center](images/launch_c4a/19.png "Select the Create button to launch the instance") -![](images/launch_c4a/18.png) +After a few seconds, your c4a instance is up and running, and you are ready to continue to the next section. -13. **Launch the instance:** Click "Create" to bring up the instance. - -![](images/launch_c4a/19.png) - -After a few seconds, your c4a instance starts up, and you are ready to continue to the next section. In the next step, you will launch the second VM, an Intel-based Emerald Rapids c4-standard-8 (c4 for short), which will serve as the comparison system for our benchmarking tests. +In the next section, you'll launch the second VM, an Intel-based Emerald Rapids c4-standard-8 (referred to as "c4"), which serves as the comparison system for benchmarking. diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md index 9f8552fbea..c747426dae 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/installing_go_and_sweet.md @@ -1,38 +1,39 @@ --- -title: Installing Go and Sweet +title: Install Go, Sweet, and Benchstat weight: 40 ### FIXED, DO NOT MODIFY layout: learningpathall --- -In this section, you'll install Go, Sweet, and the Benchstat comparison tool on both VMs. +In this section, you'll install Go, Sweet, and Benchstat on both virtual machines: -## Installation Script - -Sweet is a Go benchmarking tool that provides a standardized way to run performance tests across different systems. Benchstat is a companion tool that analyzes and compares benchmark results, helping you understand performance differences between systems. Together, these tools will allow you to accurately measure and compare Go performance on Arm and x86 architectures. +* Sweet is a Go benchmarking tool that provides a standardized way to run performance tests across systems. +* Benchstat is a companion tool that compares benchmark results to highlight meaningful performance differences. +Together, these tools help you evaluate Go performance on both Arm and x86 architectures. {{% notice Note %}} -Subsequent steps in the learning path assume you are running this script (installing) from your home directory (`$HOME`), resulting in the creation of a `$HOME/benchmarks/sweet` final install path. If you decide to install elsewhere, you will need to adjust the path accordingly when prompted to run the benchmark logic later in the learning path. +Subsequent steps in this Learning Path assume you are running this script (installing) from your home directory (`$HOME`), resulting in the creation of a `$HOME/benchmarks/sweet` final install path. If you install to a different directory, update the paths in later steps to match your custom location. {{% /notice %}} +## Installation script -Start by copying and pasting the script below on **both** of your GCP VMs. This script checks the architecture of your running VM, installs the required Go package on your VM. It then installs sweet, benchmarks, and the benchstat tools. +Start by copying and pasting the script below on both of your GCP VMs. This script automatically detects your system architecture, installs the appropriate Go version, and sets up Sweet, Benchstat, and the Go benchmark suite. -**You don't need to run it after pasting**, just paste it into your home directory and press enter to install all needed dependencies: +Paste the full block into your terminal. This creates and runs an installer script directly from your home directory: ```bash #!/usr/bin/env bash -# Write the script to filesystem using a HEREDOC +# Write the install script to filesystem using a HEREDOC cat <<'EOF' > install_go_and_sweet.sh sudo apt-get -y update sudo apt-get -y install git build-essential # Detect architecture - this allows the same script to work on both -# our Arm (c4a) and x86 (c4) VMs without modification +# Arm (c4a) and x86 (c4) VMs without modification ARCH=$(uname -m) case "$ARCH" in arm64|aarch64) @@ -90,7 +91,9 @@ chmod 755 install_go_and_sweet.sh ``` -The end of the output should look like: +## Expected output from sweet get + +When `sweet get` completes successfully, you’ll see output similar to: ```output Sweet v0.3.0: Go Benchmarking Suite @@ -109,7 +112,7 @@ Usage: sweet get [flags] ``` -## Verify Installation +## Verify installation To test that everything is installed correctly, set the environment variables shown below on each VM: diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md index 281d6bc3a8..c98ac9273c 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchmark.md @@ -1,34 +1,44 @@ --- -title: Manually running benchmarks +title: Manually run benchmarks weight: 51 ### FIXED, DO NOT MODIFY layout: learningpathall --- -In this section, you'll download the results of the benchmark you ran manually in the previous sections from each VM. You will use these results to understand how `sweet` and `benchstat` work together. +In this section, you'll download the benchmark results you ran manually in previous steps from each VM. You will use these results to understand how `sweet` and `benchstat` work together. -## Download Benchmark Results from each VM -Lets walk through the steps to manually download the sweet benchmark results from your initial run on each VM. +## Download benchmark results from each VM +Start by retrieving the results generated by Sweet from your earlier benchmark runs. -1. **Locate results:** Change directory to the `results/markdown` directory and list the files to see the `arm-benchmarks.result` file: +### Locate results + +Change directory to the `results/markdown` directory and list the files to see the `arm-benchmarks.result` file: ```bash cd results/markdown ls -d $PWD/* ``` -2. **Copy result path:** Copy the absolute pathname of `arm-benchmarks.result`. +### Copy result path + +Copy the absolute pathname of `arm-benchmarks.result`. You'll need this to initiate the download. + +### Download results + +Select `DOWNLOAD FILE` in your GCP terminal interface. Paste the absolute pathname you copied into the dialog and confirm the download. This downloads the benchmark results to your local machine. + + ![alt-text#center](images/run_manually/6.png "Download the results") -3. **Download results:** Click `DOWNLOAD FILE`, and paste the **ABSOLUTE PATHNAME** you just copied for the filename, and then click `Download`. This will download the benchmark results to your local machine. +### Rename the file - ![](images/run_manually/6.png) +After downloading the file to your local machine, rename it to `c4a.result` to distinguish it from the x86 results you'll download next. This naming convention helps you clearly identify which architecture each result came from. You'll know the download was successful if you see the file named `c4a.result` in your Downloads folder and receive a confirmation in your browser. -4. **Rename the file:** Once downloaded, on your local machine, rename this file to `c4a.result` so you can distinguish it from the x86 results you'll download later. This naming convention will help you clearly identify which results came from which architecture. You'll know the file downloaded successfully if you see the file in your Downloads directory with the name `c4a.result`, as well as the confirmation dialog in your browser: + ![alt-text#center](images/run_manually/7.png "A successful download") - ![](images/run_manually/7.png) +### Repeat for the second VM -5. **Repeat for c4 instance:** Repeat steps 2-8 with your `c4` (x86) instance. Do everything the same, except after downloading the c4's `arm-benchmarks.result` file, rename it to `c4.result`. +Repeat the same process with your c4 (x86) VM. Use the same results/markdown directory and download the `arm-benchmarks.result` file. This time, rename it to `c4.result` after downloading. -Now that you have the results from both VMs, in the next section, you'll learn how to use benchstat to analyze these results and understand the performance differences between the two architectures. +Now that you have the results from both VMs, in the next section, you'll learn how to use Benchstat to analyze these results and understand the performance differences between the two architectures. diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md index 66ff075f26..d3949feeb3 100755 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/manual_run_benchstat.md @@ -1,47 +1,55 @@ --- -title: Manually running benchstat +title: Manually run Benchstat weight: 52 ### FIXED, DO NOT MODIFY layout: learningpathall --- -You've successfully run and downloaded the benchmark results from both your Arm-based and x86-based VMs. In this section, you'll compare them to each other using the benchstat tool. +You've successfully run and downloaded the benchmark results from both your Arm-based and x86-based VMs. In this section, you'll use Benchstat to compare performance between the two instances. -## Inspecting the Results Files +## Inspect the results files -With the results files downloaded to your local machine, if you're curious to what they look like, you can inspect them to understand better what `benchstat` is analyzing. +To understand what Benchstat analyzes, open the results files to view the raw benchmark output. -1. **View raw results:** Open the `c4a.result` file in a text editor, and you'll see something like this: +Open the `c4a.result` file in a text editor. You should see something like this: - ![](images/run_manually/11.png) + ![alt-text#center](images/run_manually/11.png "A results file") The file contains the results of the `markdown` benchmark run on the Arm-based c4a VM, showing time and memory stats taken for each iteration. If you open the `c4.result` file, you'll see similar results for the x86-based c4 VM. -2. **Close the editor:** Close the text editor when done. +Close the text editor when done. -## Running Benchstat to Compare Results +## Run Benchstat to compare results -To compare the results, you'll use `benchstat` to analyze the two result files you downloaded. Since all the prerequisites are already installed on the `c4` and `c4a` instances, benchstat will be run from one of those instances. +To compare the results, you'll now use Benchstat to analyze the two result files you downloaded. Since all the prerequisites are already installed on the `c4` and `c4a` instances, Benchstat will be run from one of those instances. -1. **Create working directory:** Make a temporary benchstat directory to hold the results files on either the c4a or c4 instance, and change directory into it: +### Create working directory + +Make a temporary benchstat directory to hold the results files on either the c4a or c4 instance, and change directory into it: ```bash mkdir benchstat_results cd benchstat_results ``` -2. **Upload result files:** Click the `UPLOAD FILE` button in the GCP console, and upload the `c4a.results` AND `c4.results` files you downloaded earlier. (This uploads them to your home directory, not to the current directory.) +### Upload results files + +Click the `UPLOAD FILE` button in the GCP console, and upload the `c4a.results` AND `c4.results` files you downloaded earlier. (This uploads them to your home directory, not to the current directory.) + + ![alt-text#center](images/run_manually/16.png "Upload results file") - ![](images/run_manually/16.png) +### Verify upload -3. **Verify upload:** You'll know it worked correctly via the confirmation dialog in your terminal: +You'll know it worked correctly via the confirmation dialog in your terminal: - ![](images/run_manually/17.png) + ![alt-text#center](images/run_manually/17.png "Confirmation dialog in terminal") -4. **Move files to working directory:** Move the results files to the `benchstat_results` directory, and confirm their presence: +### Move files to working directory + +Move the results files to the `benchstat_results` directory, and confirm their presence: ```bash mv ~/c4a.results ~/c4.results . @@ -54,7 +62,9 @@ To compare the results, you'll use `benchstat` to analyze the two result files y c4.results c4a.results ``` -5. **Run benchstat:** Now you can run `benchstat` to compare the two results files: +### Run benchstat + +Now you can run `benchstat` to compare the two results files: ```bash export GOPATH=$HOME/go @@ -63,7 +73,9 @@ To compare the results, you'll use `benchstat` to analyze the two result files y benchstat c4a.results c4.results > c4a_vs_c4.txt ``` -6. **View comparison results:** Run the `cat` command to view the results: +### View comparison results + +Run the `cat` command to view the results: ```bash cat c4a_vs_c4.txt @@ -114,7 +126,7 @@ To compare the results, you'll use `benchstat` to analyze the two result files y In this example, you can see that the c4a (Arm) instance completed the markdown benchmark in 143.9m seconds, while the c4 (x86) instance took 158.3m seconds, indicating better performance on the Arm system for this particular workload. - If you wanted the results in CSV format, you could run the `benchstat` command with the `-format csv` option instead. + If you want the results in CSV format, you can run the `benchstat` command with the `-format csv` option instead. At this point, you can download the `c4a_vs_c4.txt` for further analysis or reporting. You can also run the same or different benchmarks with the same, or different combinations of VMs, and continue comparing results using `benchstat`. diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md index 4a5995608f..e4327d9c18 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/overview.md @@ -6,28 +6,32 @@ weight: 10 layout: learningpathall --- -# Go Benchmarking Overview +## Overview of Go benchmarking tools -In this section, you will learn how to measure, collect, and compare Go performance data across different CPU architectures. This knowledge is essential for developers and system architects who need to make informed decisions about infrastructure choices for their Go applications. +This section shows you how to measure, collect, and compare Go performance data across different CPU architectures. These techniques help developers and system architects make informed infrastructure decisions for their Go applications. You'll gain hands-on experience with: -- **Go Benchmarks**, a collection of pre-written benchmark definitions that standardizes performance tests for popular Go applications, leveraging Go's built-in benchmark support. +- **Go Benchmarks** - standardized definitions for popular Go applications, using Go’s built-in testing framework. -- **Sweet**, a benchmark runner that automates running Go benchmarks across multiple environments, collecting and formatting results for comparison. +- **Sweet** - a benchmark runner that automates execution and formats results for comparison across multiple environments. -- **Benchstat**, a statistical comparison tool that analyzes benchmark results to identify meaningful performance differences between systems. +- **Benchstat** - a statistical comparison tool that analyzes benchmark results to identify meaningful performance differences between systems. -Benchmarking is critical for modern software development because it allows you to: -- Quantify the performance impact of code changes -- Compare different hardware platforms objectively -- Make data-driven decisions about infrastructure investments -- Identify optimization opportunities in your applications +Benchmarking is critical for modern software development because it allows you to do the following: +- Quantify the impact of code changes +- Compare performance across hardware architectures +- Make data-driven decisions about infrastructure +- Identify optimization opportunities in your application code -You'll use Intel c4-standard-8 and Arm-based c4a-standard-4 (both four-core) instances running on GCP to run and compare benchmarks using these tools. +In this Learning Path, you'll compare performance using two four-core GCP instance types: + +* The Arm-based c4a-standard-4 +* The Intel-based c4-standard-8 {{% notice Note %}} -Arm-based c4a-standard-4 instances and Intel-based c4-standard-8 instances both utilize four cores. Both instances are categorized by GCP as members of the **consistently high performing** series; the main difference between the two is that the c4a has 16 GB of RAM, while the c4 has 30 GB of RAM. We've chosen to keep CPU cores equivalent across the two instances of the same series to keep the comparison as close as possible. +Arm-based c4a-standard-4 instances and Intel-based c4-standard-8 instances both utilize four cores. Both instances are categorized by GCP as members a series that demonstrates consistent high performance. +The main difference between the two is that c4a has 16 GB of RAM, while c4 has 30 GB of RAM. This Learning Path uses equivalent core counts to ensure a fair performance comparison. {{% /notice %}} diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md index 86882fcfd8..62302bea62 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_install.md @@ -1,58 +1,70 @@ --- -title: Installing the Automated Benchmark and Benchstat Runner +title: Install the automated benchmark and Benchstat runner weight: 53 ### FIXED, DO NOT MODIFY layout: learningpathall --- -In the last section, you learned how to run benchmarks and benchstat manually. Now you'll learn how to run them automatically, with enhanced visualization of the results. +In the last section, you learned how to run benchmarks and Benchstat manually. Now you'll automate that process and generate visual reports using a script called `rexec_sweet.py`. -## Introducing rexec_sweet.py +## What is rexec_sweet.py? -The `rexec_sweet.py` script is a powerful automation tool that simplifies the benchmarking workflow. This tool connects to your GCP instances, runs the benchmarks, collects the results, and generates comprehensive reports—all in one seamless operation. It provides several key benefits: +`rexec_sweet.py` is a script that automates the benchmarking workflow: it connects to your GCP instances, runs benchmarks, collects results, and generates HTML reports - all in one step. + +It provides several key benefits: - **Automation**: Runs benchmarks on multiple VMs without manual SSH connections - **Consistency**: Ensures benchmarks are executed with identical parameters - **Visualization**: Generates HTML reports with interactive charts for easier analysis -The only dependency you are responsible for satisfying before the script runs is completion of the "Installing Go and Sweet" sections of this learning path. Additional dependencies are dynamically loaded at install time by the install script. +Before running the script, ensure you've completed the "Install Go, Sweet, and Benchstat" step. All other dependencies are installed automatically by the setup script. + +## Set up rexec_sweet + +Follow the steps below to set up rexec_sweet.py. -## Setting up rexec_sweet +### Create a working directory -1. **Create a working directory:** On your local machine, open a terminal, then create and change into a directory to store the `rexec_sweet.py` script and related files: +On your local machine, open a terminal, then create and change into a directory to store the `rexec_sweet.py` script and related files: ```bash mkdir rexec_sweet cd rexec_sweet ``` -2. **Clone the repository inside the directory:** Get the `rexec_sweet.py` script from the GitHub repository: +### Clone the repository + +Get the `rexec_sweet.py` script from the GitHub repository: ```bash git clone https://github.com/geremyCohen/go_benchmarks.git cd go_benchmarks ``` -3. **Run the installer:** Copy and paste this command into your terminal to run the installer: +### Run the installer + +Copy and paste this command into your terminal to run the installer: ```bash ./install.sh ``` - If the install.sh script detects that you already have dependencies installed, it may ask you if you wish to reinstall them with the following prompt as shown: + If the install.sh script detects that you already have dependencies installed, it might ask you if you want to reinstall them: ```output pyenv: /Users/gercoh01/.pyenv/versions/3.9.22 already exists continue with installation? (y/N) ``` - If you see this prompt, enter `N` (not `Y`!) to continue with the installation without modifying the existing installed dependencies. + If you see this prompt, enter `N` to continue with the installation without modifying the existing installed dependencies. + +### Verify VM status -4. **Verify VM status:** Make sure the GCP VM instances you created in the previous section are running. If not, start them now, and give them a few minutes to come up. +Make sure the GCP VM instances you created in the previous section are running. If not, start them now, and wait a few minutes for them to finish booting. {{% notice Note %}} -The install script will prompt you to authenticate with Google Cloud Platform (GCP) using the gcloud command-line tool at the end of install. If after installing you have issues running the script and/or get GCP authentication errors, you can manually authenticate with GCP by running the following command: `gcloud auth login` +The install script prompts you to authenticate with Google Cloud Platform (GCP) using the gcloud command-line tool at the end of install. If after installing you have issues running the script and/or get GCP authentication errors, you can manually authenticate with GCP by running the following command: `gcloud auth login` {{% /notice %}} diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md index b2cfbf4ba5..67be9e6393 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/rexec_sweet_run.md @@ -1,22 +1,22 @@ --- -title: Running the Automated Benchmark and Benchstat Runner +title: Run the automated benchmark and Benchstat runner weight: 54 ### FIXED, DO NOT MODIFY layout: learningpathall --- +With `rexec_sweet` installed, your benchmarking instances running, and your local machine authenticated with GCP, you're ready to run automated benchmarks across your configured environments. -With `rexec_sweet` installed, your benchmarking instances running, and your localhost authenticated with GCP, you'll now see how to run benchmarks in an automated fashion. +## Run an automated benchmark and generate results -## Run an Automated Benchmark and Analysis - -1. **Run the script:** Execute the `rexec_sweet` script from your local terminal: +To begin, open a terminal on your local machine and run: ```bash rexec_sweet ``` +The script will prompt you to choose a benchmark. -2. **Select a benchmark:** The script will prompt you for the name of the benchmark you want to run. Press enter to run the default benchmark, which is `markdown` (this is the recommended benchmark to run the first time.) +Press **Enter** to run the default benchmark, markdown, which is a good starting point for your first run. ```bash Available benchmarks: @@ -33,7 +33,7 @@ Available benchmarks: Enter number (1-10) [default: markdown]: ``` -3. **Select instances:** The script will proceed and call into GCP to detect all running VMs. You should see the script output: +The script then detects your running GCP instances and displays them. You’ll be asked whether you want to use the first two instances it finds and the default install paths. ```output Available instances: @@ -42,14 +42,11 @@ Available instances: Do you want to run the first two instances found with default install directories? [Y/n]: ``` +You can accept the defaults by pressing **Enter**, which uses the instances listed and assumes Go and Sweet were installed to ~/benchmarks/sweet. -4. **Choose your configuration:** You have two options: - - - **Use default settings:** If you want to run benchmarks on the instances labeled with "will be used as nth instance", and you installed Go and Sweet into the default directories as noted in the tutorial, you can press Enter to accept the defaults. +If you're running more than two instances or installed Go and Sweet to a non-default location, enter n and follow the prompts to manually select instances and specify custom install paths. - - **Custom configuration:** If you are running more than two instances, and the script doesn't suggest the correct two to autorun, or you installed Go and Sweet to non-default folders, select "n" and press Enter. The script will then prompt you to select the instances and runtime paths. - -In this example, we'll manually select the instances and paths as shown below: +In this example, you'll manually select the instances and paths as shown below: ```output Available instances: @@ -73,9 +70,9 @@ Output directory: /private/tmp/a/go_benchmarks/results/c4-c4a-markdown-20250610T ... ``` -Upon entering instance names and paths for the VMs, the script will automatically: - - Run the benchmark on both VMs - - Run `benchstat` to compare the results +After selecting instances and paths, the script will: + - Run the selected benchmark on both VMs + - Use `benchstat` to compare the results - Push the results to your local machine ```output @@ -88,15 +85,17 @@ Running benchmarks on the selected instances... Report generated in results/c4-c4a-markdown-20250610T190407 ``` -5. **View the report:** Once on your local machine, `rexec_sweet` will generate an HTML report that will open automatically in your web browser. +### View the report + +Once on your local machine, `rexec_sweet` will generate an HTML report that opens automatically in your web browser. - If you close the tab or browser, you can always reopen the report by navigating to the `results` subdirectory of the current working directory of the `rexec_sweet.py` script, and opening `report.html`. + If you close the report, you can reopen it by navigating to the `results` subdirectory and opening report.html in your browser. -![](images/run_auto/2.png) +![alt-text#center](images/run_auto/2.png "Sample HTML report") {{% notice Note %}} -If you see output messages from `rexec_sweet.py` similar to "geomeans may not be comparable" or "Dn: ratios must be >0 to compute geomean", this is expected and can be ignored. These messages indicate that the benchmark sets differ between the two VMs, which is common when running benchmarks on different hardware or configurations. +If you see output messages from `rexec_sweet.py` similar to "geomeans may not be comparable" or "Dn: ratios must be >0 to compute geomean", this is expected and can be ignored. These warnings typically appear when benchmark sets differ slightly between the two VMs. {{% /notice %}} -6. **Analyze results:** Upon completion, the script will generate a report in the `results` subdirectory of the current working directory of the `rexec_sweet.py` script, which opens automatically in your web browser to view the benchmark results and comparisons. +Upon completion, the script generates a report in the `results` subdirectory of the current working directory of the `rexec_sweet.py` script, which opens automatically in your web browser to view the benchmark results and comparisons. diff --git a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md index 8ddf05bec3..8687181e66 100644 --- a/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md +++ b/content/learning-paths/servers-and-cloud-computing/go-benchmarking-with-sweet/running_benchmarks.md @@ -1,24 +1,25 @@ --- -title: Benchmark Types and Metrics +title: Benchmark types and metrics weight: 50 ### FIXED, DO NOT MODIFY layout: learningpathall --- -With setup complete, you can now run and analyze the benchmarks. Before you do, it's good to understand all the different pieces in more detail. +Now that setup is complete, it's important to understand the benchmarks you’ll run and the performance metrics you’ll use to evaluate results across systems. -## Choosing a Benchmark to Run +## Available benchmarks Whether running manually or automatically, the benchmarking process consists of two main steps: -1. **Running benchmarks with Sweet**: `sweet` executes the benchmarks on each VM, generating raw performance data +1. **Running benchmarks with Sweet**: `sweet` executes the benchmarks on each VM, generating raw performance data. 2. **Analyzing results with Benchstat**: `benchstat` compares the results from different VMs to identify performance differences. Benchstat can output results in text format (default) or CSV format. The text format provides a human-readable tabular view, while CSV allows for further processing with other tools. Sweet comes ready to run with the following benchmarks: -| Benchmark | Description | Command | + +| Benchmark | Description | Example command | |-----------------|-------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------| | **biogo-igor** | Processes pairwise alignment data using the biogo library, grouping repeat feature families and outputting results in JSON format. | `sweet run -count 10 -run="biogo-igor" config.toml` | | **biogo-krishna** | Pure-Go implementation of the PALS algorithm for pairwise sequence alignment, measuring alignment runtime performance. | `sweet run -count 10 -run="biogo-krishna" config.toml` | @@ -31,18 +32,18 @@ Sweet comes ready to run with the following benchmarks: | **markdown** | Parses and renders Markdown documents to HTML using a Go-based markdown library to evaluate parsing and rendering throughput. | `sweet run -count 10 -run="markdown" config.toml` | | **tile38** | Stress-tests a Tile38 geospatial database with WITHIN, INTERSECTS, and NEARBY queries to measure spatial query performance. | `sweet run -count 10 -run="tile38" config.toml` | -## Metrics Summary +## Performance metrics When running benchmarks, several key metrics are collected to evaluate performance. The following summarizes the most common metrics and their significance: -### Seconds per Operation - Lower is better +### Seconds per operation (lower is better) This metric measures the time taken to complete a single operation, indicating the raw speed of execution. It directly reflects the performance efficiency of a system for a specific task, making it one of the most fundamental benchmarking metrics. A system with lower seconds per operation completes tasks faster. This metric primarily reflects CPU performance but can also be influenced by memory access speeds and I/O operations. If seconds per operation is the only metric showing significant difference while memory metrics are similar, the performance difference is likely CPU-bound. -### Operations per Second - Higher is better +### Operations per second (higher is better) This metric provides a clear measure of system performance capacity, making it essential for understanding raw processing power and scalability potential. A system performing more operations per second has greater processing capacity. This metric reflects overall system performance including CPU speed, memory access efficiency, and I/O capabilities. @@ -51,7 +52,7 @@ If operations per second is substantially higher while memory usage remains prop This metric is essentially the inverse of "seconds per operation" and provides a more intuitive way to understand throughput capacity. -### Average RSS Bytes - Lower is better +### Average RSS bytes (lower is better) Resident Set Size (RSS) represents the portion of memory occupied by a process that is held in RAM (not swapped out). It shows the typical memory footprint during operation, indicating memory efficiency and potential for scalability. @@ -59,7 +60,7 @@ Lower average RSS indicates more efficient memory usage. A system with lower ave If one VM has significantly higher seconds per operation but lower RSS, it may be trading speed for memory efficiency. Systems with similar CPU performance but different RSS values indicate different memory optimization approaches; lower RSS with similar CPU performance suggests better memory management, which is a critical indicator of performance in memory-constrained environments. -### Peak RSS Bytes - Lower is better +### Peak RSS bytes (lower is better) Peak RSS bytes is the maximum Resident Set Size reached during execution, representing the worst-case memory usage scenario. The peak RSS metric helps to understand memory requirements and potential for memory-related bottlenecks during intensive operations. @@ -67,7 +68,7 @@ Lower peak RSS indicates better handling of memory-intensive operations. High pe Large differences between average and peak RSS suggest memory usage volatility. A system with lower peak RSS but similar performance is better suited for memory-constrained environments. -### Peak VM Bytes - Lower is better +### Peak VM bytes (lower is better) Peak VM Bytes is the maximum Virtual Memory size used, including both RAM and swap space allocated to the process. @@ -75,7 +76,7 @@ Lower peak VM indicates more efficient use of the total memory address space. Hi If peak VM is much higher than peak RSS, the system is relying heavily on virtual memory management. Systems with similar performance but different VM usage patterns may have different memory allocation strategies. High VM with performance degradation suggests potential memory-bound operations due to excessive paging. -## Summary of Efficiency Indicators +## Summary of efficiency indicators When comparing metrics across two systems, keep the following in mind: @@ -84,24 +85,24 @@ A system is likely CPU-bound if seconds per operation differs significantly whil A system is likely memory-bound if performance degrades as memory metrics increase, especially when peak RSS approaches available physical memory. -### Efficiency Indicators +### Efficiency indicators The ideal system shows lower values across all metrics - faster execution with smaller memory footprint. Systems with similar seconds per operation but significantly different memory metrics indicate different optimization priorities. -### Scalability Potential +### Scalability potential Lower memory metrics (especially peak values) suggest better scalability for concurrent workloads. Systems with lower seconds per operation but higher memory usage may perform well for single tasks but scale poorly. -### Optimization Targets +### Optimization targets Large gaps between average and peak memory usage suggest opportunities for memory optimization. High seconds per operation with low memory usage suggests CPU optimization potential. -## Best Practices when benchmarking across different instance types +## Best practices when benchmarking across different instance types Here are some general tips to keep in mind as you explore benchmarking across different apps and instance types: -- Unlike Intel and AMD processors that use hyper-threading, Arm processors provide single-threaded cores without hyper-threading. A four-core Arm processor has four independent cores running four threads, while an four-core Intel processor provides eight logical cores through hyper-threading. This means each Arm vCPU represents a full physical core, while each Intel/AMD vCPU represents half a physical core. For fair comparison, this learning path uses a 4-vCPU Arm instance against an 8-vCPU Intel instance. When scaling up instance sizes during benchmarking, make sure to keep a 2:1 Intel/AMD:Arm vCPU ratio if you wish to keep parity on CPU resources. +- Unlike Intel and AMD processors that use hyper-threading, Arm processors provide single-threaded cores without hyper-threading. A four-core Arm processor has four independent cores running four threads, while a four-core Intel processor provides eight logical cores through hyper-threading. This means that each Arm vCPU represents a full physical core, while each Intel/AMD vCPU represents half a physical core. For fair comparison, this Learning Path uses a 4-vCPU Arm instance against an 8-vCPU Intel instance. When scaling up instance sizes during benchmarking, make sure to keep a 2:1 Intel/AMD:Arm vCPU ratio if you wish to keep parity on CPU resources. -- It's suggested to run each benchmark at least 10 times (specified via the `count` parameter) to handle outlier/errant runs and ensure statistical significance. +- Run each benchmark at least 10 times (-count 10) to account for outliers and produce statistically meaningful results. -- Results may be bound by CPU, memory, or I/O performance. If you see significant differences in one metric but not others, it may indicate a bottleneck in that area; running the same benchmark with different configurations (e.g., more CPU cores, more memory) can help identify the bottleneck. +- Results can be bound by CPU, memory, or I/O performance. If you see significant differences in one metric but not others, it might indicate a bottleneck in that area; running the same benchmark with different configurations (for example, using more CPU cores or more memory) can help identify the bottleneck.