How To Work With This Repository

Within this repository, you'll discover the source code, detailed instructions, and, where applicable, the results generated from my research on the RAST approach that was originally introduced at the MASCOTS 2022 conference.

To effectively navigate the historical states of this repository corresponding to the specific papers, I've organized them into dedicated branches. For instance, the "MASCOTS2022" Branch represents the exact state of the repository at the time of crafting and presenting the paper submitted for review and later published at MASCOTS 2022. After selecting the desired Branch, look for the appropriate conference name in the README file.

Improving RAST with SDN (WIP)

Last change from: 05.01.2024

Currently, I am working on improving the prediction quality of RAST by enriching the request logs with network metrics recorded by an SDN controller. For my experiments, I use containernet (fork of mininet) as the network emulator and pox as the SDN controller. I built a containernet topology that encapsulates two different kinds of experiments:

1. Running an experiment using the TeaStore benchmarking application (our TeaStore Fork).
2. Running an experiment with a simulated version of TeaStore.

Within an experiment a load test is used to generate request logs and network metrics.

Preparations:

• Clone pox in a directory called `pox. Use the halosaur branch.
• Clone the locust_scripts repository in a directory called locust_scripts.
• Clone our TeaStore Fork repository to your local machine.
• Build the TeaStore application and the associated Docker images by following the instructions provided by the developers. You can find these instructions in the GET_STARTED.md file within the cloned repository.
• Both directories should be on the same directory level, e.g., ~/pox and ~/locust_scripts.

Recommended workflow (for terminal users):

• This session will be used to start the Load Test. Navigate to the locust_scripts folder.
• sudo -s to gain root access (required for containernet).
• source activate_venv.sh
• (python mininet/teastore_topo.py --help) to show help message.
• python mininet/teastore_topo.py to run one workload intensity,
• python mininet/teastore_topo.py -a to run all,
• -s uses the simulation for the experiment, e.g., python mininet/teastore_topo.py -s -a

The relevant files after the experiment are:

• switch_flow_stats_{date}.json: contains the observed bytes and packets per second per SDN switch.
• TeaStore kieker logs (needs to be downloaded from the /logs endpoint of TeaStore): contains the kieker logs of the real TeaStore application. Needs to be converted using our Kieker_ETL project to be usable for RAST.
• Simulators/teastore_simulation.log: contains the request logs of our simulator including received requests and their processing times, already in a usable format for RAST.
• locust_log.log: contains the logs of the load tester, requests and their response times.

These files serve as input to our different analysis tools in the Regression-Analysis_Workload-Characterization project.

Improved RAST Simulator and Load Tester Components (PADS 2024)

Last change from: 29.04.2024

TODO: Accepted for publication at PADS 2024. Add link.

In our paper, we explain, implement and evaluate improvements to the following components of RAST:

• Predictive Model Creator,
• Simulator,
• Load Tester.

For our evaluation, we use the TeaStore benchmarking application. We created a GitHub fork and modified some parameters of their logging framework to create consistent logs especially for higher intensity workloads.

This repository contains the improved Load Tester whereas the Simulator and the predictive models we created can be found in our Simulators repository. The Automated Estimator Pipeline (see below) is one implementation of the remaining RAST components (including the Predictive Model Creator).

Additional experimental results not published in our paper, including the Python scripts we use for similarity calculation, can be accessed in our Datalore notebook.

Instructions for Conducting Experiments

To conduct the experiments outlined in this paper, please follow the step-by-step instructions below:

Preparations:

• Clone our TeaStore Fork repository to your local machine.
• Build the TeaStore application and the associated Docker images by following the instructions provided by the developers. You can find these instructions in the GET_STARTED.md file within the cloned repository.
• Clone our Simulators repository to your local machine.
• Clone the locust_scripts repository.

Recommended workflow (for terminal users):

1. Open your terminal and use a terminal multiplexer such as tmux to create four sessions. We will refer to these sessions by numbers:
   • Session (1): This session will be used to start the TeaStore or the Simulator. Navigate to the respective folder within the cloned repositories.
   • Session (2): This session will be used to start the Load Test. Navigate to the locust_scripts folder.
   • Session (3): This session will be used to make code changes to the offical_teastore_locustfile.py file, allowing you to modify the load intensity profile. Navigate to the locust_scripts/locust folder and open the file using a text editor of your choice (e.g., Vim or Emacs).
   • Session (4): This session will be used to make code changes to the teastore.kt file, enabling you to modify the predictive model. Navigate to the Simulators folder and open the file.
2. In Session (1), start the TeaStore or the Simulator based on the measurements you wish to acquire. For the purpose of this explanation, we will focus on starting the Simulator. Navigate to your local Simulator folder and execute the command ./gradlew run. If successful, you will see the following line printed on the console: INFO ktor.application - Responding at http://0.0.0.0:8081. To terminate the Simulator, press Ctrl + C.
3. In Session (2):
   1. (Recommended):
      1. Create a python virtual environment in a directory called venv, e.g., python3 -m venv venv
      2. Run the command source activate_venv.sh to activate the Python virtual environment (venv).
   2. Execute ./start_teastore_loadtest.sh to initiate the load test. This repository uses a low load intensity by default. The load test will automatically conclude after approximately two minutes.
   3. Clean the folder by executing ./delete_results.sh.
4. In Session (4), you can now examine the teastore_simulation.log file. This file contains simulated processing times generated by the predictive model, among other relevant information.
5. To modify the load intensity profile, navigate to Session (3) and locate the StagesShape class within the offical_teastore_locustfile.py file. Look for the line load_intensity_profile: LoadIntensityProfile = LoadIntensityProfile.LOW. Set load_intensity_profile to your desired value.
6. To modify the predictive model, navigate to Session (4) and follow the instructions in the README.md file within the Simulators repository.

Automated Estimator Pipeline (SIMUtools 2023)

Last update from: 03.06.2023

DOI: https://doi.org/10.1007/978-3-031-57523-5_2

The Automated Estimator Pipline by Adrian Liermann was developed as part of his master's thesis. In his thesis, he implements and evaluates improvements to our RAST approach, in particular, improvements to the training process. In the RAST approach, the following components are of particular interest to the training process:

• Log Transformer
• Predictive Model Creator

The Automated Estimator Pipeline is an implementation of these improved components wrapped in a CLI tool.

RAST Simulator and Load Tester Components (MASCOTS 2022)

Last update from: 25.06.2023

In our latest publication at MASCOTS 2022, we explain our extension of this project.

• We provide a regression model learned from log files using our RAST approach;
• we improved the Simulator so that it uses the regression model to simulate processing times of the System-Under-Evaluation (SUE);
• we built a mininet topology to ease the process of reproducibly launching experiments. We modelled the link parameters based on the infrastructure of the SUE;
• we implemented additional ancillary Python scripts to help analyse the log files.

Instructions to launch an experiment

Quick start

Tested with

• Python 3.8.10 on Ubuntu 20.04 LTS
• Python 3.9.12 on macOS 12.6.1

• Clone the MASCOTS2022 branch of this repository
• run cd <the_directory_you_cloned_the_repository>
• create a python virtual environment in a directory called venv, e.g., python3 -m venv venv
• activate virtual environment with source activate_venv.sh
• run pip install -r requirements.txt
• install mininet using the command sudo apt-get install mininet (in case you need additional help, consult the mininet documentation
• run cd mininet
• execute ./start_mininet.sh
• if mininet successfully started the experiment, you will notice a series of log files that were created:
  • ARS_simulation.log (means that the Simulator is running)
  • locust-parameter-variation.log (means that the load tester is generating the alarm device workload)
  • locust.log (means that the load tester is generating the background workload)
• in another terminal, run cd <the_directory_you_cloned_the_repository>
• run tail -f locust-parameter-variation.log
• the experiment may run for a couple of hours. The experiment ends, once the locust-parameter-variation.log prints the message "Finished performance test. System failed at ..."
• after that, you can analyse the log files. The locust-parameter-variation.log is of particular interest as it contains the measured average and maximum response times depending on the number of simulated alarm devices.

Locust Performance Testing Infrastructure (MASCOTS 2020)

Last update from: 25.06.2023

In [1], we introduced a generic performance testing infrastructure and used it in an industrial case study. Our idea is to have decoupled components, Python scripts in our case, that together allow to:

1. reproducible execute a load testing tool with a set of parameters for a particular experiment,
2. evaluate the performance measurements assisted by visualizations or automatic evaluators.

Generally, we have four types of components in our infrastructure:

• Executors: execute a particular Load Tester as long as the Load Tester provides a CLI or an API;
• Load Testers: execute the load test, parametrized with values given by an Executor. Have to output a logfile containing the response times;
• Evaluators: postprocess the logfile and for example plot the response times;
• Systems under Test (SUTs): Target systems we want to test. Usually, the target systems will be external systems, e.g., web servers. In our case, we build software that simulates the behavior of a real system, in order to provide the means for others to roughly reproduce our experiments.

More details about our generic performance testing infrastructure can be found in our paper [1].

This repository contains the aforementioned Python scripts:

• Executors:
  • executor.py: executes Locust with a set of parameters;
  • locust-parameter-variation.py: executes Locust and keeps increasing the load. This is similar to Locust's Step Load Mode, however, our approach increases the number of clients for as long as the ARS complies with real-time requirements in order to find the saturation point of the ARS.
• Load Testers:
  • locust_tester.py: contains specific code for Locust to perform the actual performance test. For demonstration purposes, this script tests ARS_simulation.py. Outputs a locust_log.log;
  • locust_multiple_requests: an enhanced version of locust_tester that sends additional requests to generate more load.
  • locust_teastore.py: performs load testing against TeaStore, or our simulated TeaStore.
• Evaluators:
  • loadtest_plotter.py: reads the locust_log.log, plots response times, and additional metrics to better visualize, if the real-time requirements of the EN 50136 are met.
• SUTs
  • Alarm Receiving Software Simulation (ARS_simulation.py): simulates an industrial ARS based on data measured in the production environment of the GS company group.
  • TeaStore (teastore_simulation.py): simulates TeaStore based on a predictive model generated in a lab environment.

Instructions to reproduce results in our paper

Quick start

• Clone the MASCOTS2020 branch of this repository;
• run pip3 install -r requirements.txt;
• In the file ARS_simulation.py make sure that the constant MASCOTS2020 is set to True.
• open two terminal shells:
  1. run python3 ARS_simulation.py in one of them;
  2. run python3 executor.py. in the other.
• to stop the test, terminate the executor.py script;
• run python3 loadtest_plotter.py, pass the locust_log.log and see the results. :)

Details

Using the performance testing infrastructure available in this repository, we conducted performance tests in a real-world alarm system provided by the GS company. To provide a way to reproduce our results without the particular alarm system, we build a software simulating the Alarm Receiving Software. The simulation model uses variables, we identified as relevant and also performed some measurements in the production environment, to initialize the variables correctly.

To reproduce our results, follow the steps in the Section "Quick start". The scripts are already preconfigured, to simulate a realistic workload, inject faults, and automatically recover from them. The recovery is performed after the time, the real fault management mechanism requires.

If you follow the steps and, for example, let the test run for about an hour, you will get similar results to the ones you can find in the Folder "Tests under Fault".

Results after running our scripts for about an hour:

---

Keep in mind that we use a simulated ARS here; in our paper we present measurements performed with a real system, thus the results reproduced with the code here are slightly different.

Nonetheless, the overall observations we made in our paper, are in fact reproducible.

---

Instructions on how to adapt our performance testing infrastructure to other uses

After cloning the repository, take a look at the locust_tester.py. This is, basically, an ordinary Locust script that sends request to the target system and measures the response time, when the response arrives. Our locust_tester.py is special, because:

• we implemented a custom client instead of using the default;
• we additionally log the response times to a logfile instead of using the .csv files Locust provides.

So, write a performance test using Locust, following the instructions of the Locust developers on how to write a Locust script. The only thing to keep in mind is, that your Locust script has to output the measured response times to a logfile in the same way our script does it. Use logger.info("Response time %s ms", total_time) to log the response times.

When you have your Locust script ready, execute it with python3 executor.py, pass the path to your script as argument, and when you want to finish the load test, terminate it with Ctrl + C.

Use python3 executor.py --help to get additional information.

Example call:

% python3 executor.py locust_scripts/locust_tester.py

After that, plot your results:

% python3 loadtest_plotter.py
Path to the logfile: locust_log.log