Performance Testing distGatling

Performance Testing Using Distributed Gatling

The goal of this module is to set up a performance test rig with a Multi-Region Hybrid installation and distGatling https://github.com/Abiy/distGatlingproject.

The test we are going to run will be to achive 600TPS peak load while generating traffic from 3 different regions and observing traffic flow based on geographic proximity.

The instructions are authomation-friendly. Ie., we run them step-by-step, but they are expected to be used within CI/CD context. We are using ansible as main automation tool.

The test rig is located in a different GCP project. The master node is used as ansible control box. The worker nodes are created in different regions.

---

Gatling cluster project: hybrid-emea-cs-demo2

Layout: 4 nodes; 0 -- master; 1,2,3 -- workers

Environment Variable:

export PROJECT=emea-cs-hybrid-demo2
export GW_IMAGE=centos-8-v20200326
export GW_MACHINE_TYPE=n1-standard-2
export GW_NODES=3
export GW_ZONES=( europe-west1-c europe-west1-c us-east1-c asia-east1-c )

Create cluster nodes

NOTE: if preemptible nodes are desired, add: --preemptible

From your CloudShell, execute following shell fragment:

for NODE in $( seq 0 $GW_NODES ); do
    export GW_NODE=gw-$NODE
    gcloud compute instances create $GW_NODE --project=$PROJECT --zone=${GW_ZONES[$NODE]} --machine-type=$GW_MACHINE_TYPE --image-project=centos-cloud --image=$GW_IMAGE 
done

Configure distGatling Cluster

Init ssh-agent and ssh into a GCP instance with agent forwarding (-A)

eval `ssh-agent`
ssh-add ~/.ssh/google_compute_engine
gcloud compute ssh gw-0 --project=$PROJECT --zone=${GW_ZONES[0]} --ssh-flag="-A"

WARNING: /opt/gatling/_downloads should exist!

# from localhost to @gw-0
gcloud compute scp ~/_downloads/jdk-8u211-linux-x64.rpm  yuriyl@gw-0:/opt/gatling/_downloads --project=$PROJECT --zone=${GW_ZONES[0]}

Master/Worker Layout with Populated IP Address

node	zone	private IP address
gw-0	europe-west1-c	10.132.0.45
gw-1	europe-west1-c	10.132.0.46
gw-2	us-east1-c	10.142.0.20
gw-3	asia-east1-c	10.140.0.18

?. at Gatling cluster master, install and configure Ansible

# ansible @gw-0
sudo yum -y install epel-release
sudo yum -y install ansible

?. Configure ansible and inventory

mkdir ~/ansible
cat <<EOT >>~/.ansible.cfg
[defaults]
inventory = ~/ansible/hosts
EOT

cat <<EOT >>~/ansible/hosts
# gc - gatling cluster
[gc]
gw-0
gw-1
gw-2
gw-3

# gm - gatling master
[gm]
gw-0

# gw - gatling worker
[gw]
gw-1
gw-2
gw-3
EOT

?. Define working directory structure

ansible gc -b -a "mkdir /opt/gatling"
ansible gc -b -a "sudo chown $USER: /opt/gatling"

ansible gc -a "mkdir /opt/gatling/_downloads"

?. Download Gatling

ansible gc -m shell -a "cd /opt/gatling/_downloads; curl https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/3.3.1/gatling-charts-highcharts-bundle-3.3.1-bundle.zip -O"

?. Install dependency utilities

# @gw-0
ansible gc -b -a "yum -y install zip unzip git mc"

ansible gc -b -a "yum -y install maven"

?. Download and configure JDK

NOTE: distGatling is developed with Oracle JDK 1.8. By default, Oracle JDK bundles Java FX component, used to render UI. You can of course, use Open JDK, but then it is your responcibility to add Java FX to it.

?. Copy Oracle JDK to master node.

# gw-0 for a worker node
# scp /opt/gatling/_downloads/jdk-8u211-linux-x64.rpm @gw-1:/opt/gatling/_downloads

?. Distribute JDK to worker nodes

ansible gw -m copy -a "src=/opt/gatling/_downloads/jdk-8u211-linux-x64.rpm dest=/opt/gatling/_downloads"

ansible gc -b -a "yum -y install /opt/gatling/_downloads/jdk-8u211-linux-x64.rpm"

?. During build and executing of distGatling, the code should have access to JDK with Java FX. When we installed Maven using yum, Maven installed OpenJDK 1.8, which became default and best choice for alternatives. This is sad and we now need to override it via alternatives reconfiguration and/or explicit JAVA_HOME/PATH environment variables.

java -version
sudo alternatives --config java
sudo alternatives --config javac

export JAVA_HOME=/usr/java/jdk1.8.0_211-amd64
export PATH=\$JAVA_HOME/bin:\$PATH

?. Download and configure Gatling

ansible gc -a "unzip -o /opt/gatling/_downloads/gatling-charts-highcharts-bundle-3.3.1-bundle.zip -d /opt/gatling"

ansible gc -a "ln -s /opt/gatling/gatling-charts-highcharts-bundle-3.3.1 /opt/gatling/gatling"


ansible gc -m shell -a "cd /opt/gatling;
git clone https://github.com/Abiy/distGatling.git"

ansible gc -a "mkdir /opt/gatling/gatling-workspace"

NOTE: We need to re-configure some default values in distGatling configuratiom files to adjust them to our environment and directory structure. This sidebar illustrates what we are changing. Following set of sed commends exectutes the substitution itself.

vi /opt/gatling/distGatling/gatling-rest/src/main/resources/application.yml

For master node:

# from: 
job:
  path: "/Users/ahailem/workspace/gatling/gatling-charts-highcharts-bundle-3.0.2" # Path to the base directory where the gatling lib, simulation, data, and conf are stored
  logDirectory: "/Users/ahailem/workspace/gatling/gatling-charts-highcharts-bundle-3.0.2/" # Base directory for logfiles(log/error and log/std)
  command: "/bin/bash" # Base command to run gatling.sh file
  artifact: "/Users/ahailem/workspace/gatling/gatling-charts-highcharts-bundle-3.0.2/bin/{0}.sh" # Path for the location of gatling.sh

file:
  repository: "/Users/ahailem/workspace/uploads/" # Base directory used as a temporary staging area for user file uploads(simulation files,conf,data files and lib files)

#to: 

job:
  path: "/opt/gatling/gatling" # Path to the base directory where the gatling lib, simulation, data, and conf are stored
  logDirectory: "/opt/gatling/gatling-workspace/logs" # Base directory for logfiles(log/error and log/std)
  command: "/bin/bash" # Base command to run gatling.sh file
  artifact: "/opt/gatling/gatling/bin/{0}.sh" # Path for the location of gatling.sh

file:
  repository: "/opt/gatling/gatling-workspace/uploads/" # Base directory used as a temporary staging area for user file uploads(simulation files,conf,data files and lib files)

In addition, for each worker node:

vi /opt/gatling/distGatling/gatling-agent/src/main/resources/application.yml
# from:
  jobDirectory: /Users/ahailem/workspace/gspace/ # directory to store artifacts temporarily, only applicable for agents

# to:
  jobDirectory: /opt/gatling/gatling-workspace/gspace/ # directory to store artifacts temporarily, only applicable for agents

Using sed to execute substitutions

ansible gw-1 -m shell -a '
export GATLING_HOME=/opt/gatling/gatling
sed -i -r 's%(^ +)path: "(.+)"%\1path: "'$GATLING_HOME'"%' /opt/gatling/distGatling/gatling-rest/src/main/resources/application.yml

sed -i -r 's%(^ +)logDirectory: "(.+)"%\1logDirectory: "'$GATLING_HOME'-workspace/logs"%' /opt/gatling/distGatling/gatling-rest/src/main/resources/application.yml

sed -i -r 's%(^ +)artifact: "(.+)"%\1artifact: "'$GATLING_HOME'/bin/{0}.sh"%' /opt/gatling/distGatling/gatling-rest/src/main/resources/application.yml

sed -i -r 's%(^ +)repository: "(.+)"%\1repository: "'$GATLING_HOME'-workspace/upload"%' /opt/gatling/distGatling/gatling-rest/src/main/resources/application.yml
EOF

export SCRIPT=$(cat <<EOT
export GATLING_HOME=/opt/gatling/gatling

for F in /opt/gatling/distGatling/gatling-rest/src/main/resources/application.yml /opt/gatling/distGatling/gatling-agent/src/main/resources/application.yml; do 

sed -i -r 's%(^ +)path: "(.*)"%\1path: "'\$GATLING_HOME'"%' \$F

sed -i -r 's%(^ +)logDirectory: "(.*)"%\1logDirectory: "'\$GATLING_HOME'-workspace/logs"%' \$F

sed -i -r 's%(^ +)artifact: "(.*)"%\1artifact: "'\$GATLING_HOME'/bin/{0}.sh"%' \$F

sed -i -r 's%(^ +)repository: "(.*)"%\1repository: "'\$GATLING_HOME'-workspace/upload"%' \$F

sed -i -r 's%(^ +)jobDirectory: ([^ ]*) %\1jobDirectory: "'\$GATLING_HOME'-workspace/gspace" %' \$F

done
EOT
)

ansible gc -m shell -a "$SCRIPT"

Build distGatling at all cluster nodes

export SCRIPT=$(cat <<EOT
# maven build
export JAVA_HOME=/usr/java/jdk1.8.0_211-amd64
export PATH=\$JAVA_HOME/bin:\$PATH

cd /opt/gatling/distGatling
mvn clean package

EOT
)
ansible gc -m shell -a "$SCRIPT"

Repeat for each worker node: 1..3

# @gw-0
ansible gm -m shell -a "export JAVA_HOME=/usr/java/jdk1.8.0_211-amd64;
export PATH=\$JAVA_HOME/bin:\$PATH; cd /opt/gatling/distGatling/gatling-rest;nohup /bin/bash master.sh -Dmaster.port=2551 -Dserver.port=8080 &"

# @gw-[1..3]
ansible gw -m shell -a "export JAVA_HOME=/usr/java/jdk1.8.0_211-amd64;
export PATH=\$JAVA_HOME/bin:\$PATH; cd /opt/gatling/distGatling/gatling-agent; nohup /bin/bash agent.sh -Dakka.contact-points=10.132.0.45:2551 -Dactor.port=0 -Dserver.port=8090 &"

distGatling Scenario run

?. Using distGatling UI http://:8080, Deploy following scenario as ahr.PingSimulation test.

NOTE: Assuming by-pass for a DNS configuration

package ahr

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class PingSimulation extends Simulation { 

  val httpProtocol = http 
    .baseUrl("https://emea-cs-hybrid-demo6-test.hybrid-apigee.net")
    .acceptHeader("text/html,application/json,application/xml;q=0.9,*/*;q=0.8")
    .doNotTrackHeader("1")
    .acceptLanguageHeader("en-US,en;q=0.5")
    .acceptEncodingHeader("gzip, deflate")
    .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
    .hostNameAliases(Map("emea-cs-hybrid-demo6-test.hybrid-apigee.net" -> "35.190.60.112"))

  val scn = scenario("PingSimulation")
    .exec(http("request_1") 
      .get("/ping"))
    .pause(1) 

  setUp(  
   scn.inject(
    nothingFor(4 seconds),
    atOnceUsers(10),
    rampUsers(10) during (5 seconds),
    constantUsersPerSec(20) during (15 seconds),
    constantUsersPerSec(20) during (15 seconds) randomized,
    rampUsersPerSec(10) to 20 during (10 minutes),
    rampUsersPerSec(10) to 20 during (10 minutes) randomized,
    heavisideUsers(1000) during (20 seconds)
   ).protocols(httpProtocol)
  )
}

Gatling/distGatling Test SDLC

As you develop a Gatling test, Run test

set JAVA_HOME=c:\Program Files\Java\jdk-13.0.2
set GATLING_HOME=C:\work-gatling\gatling-charts-highcharts-bundle-3.3.1
set PATH=%JAVA_HOME\bin;%GATLING_HOME%\bin;%PATH%

gatling -sf %CD%\simulations -rf %CD%\results -bf %CD%\target -s ahr.PingSimulation

Gatling Cluster Auxillary Operations

Check Status

gcloud compute instances list --project $PROJECT |grep gw-|sort

Control the cluster

# start|stop|delete
export ACTION=stop
for NODE in $( seq 0 $GW_NODES ); do
    export   GW_NODE=gw-$NODE
    gcloud compute instances $ACTION $GW_NODE --project=$PROJECT --zone=${GW_ZONES[$NODE]} --quiet
done

Curl client request example

curl https://emea-cs-hybrid-demo6-test.hybrid-apigee.net 

# with resolve
curl --resolve "emea-cs-hybrid-demo6-test.hybrid-apigee.net:443:35.190.60.112" https://emea-cs-hybrid-demo6-test.hybrid-apigee.net/ping