TechXchange 2024 Lab 1482 - JIT Server
This repository contains artifacts to demonstrate some of the benefits of Semeru Cloud Compiler (aka OpenJ9 JITServer). The architecture of the experiment is shown in this picture:
The application used as a benchmark is AcmeAirEE8, a Java EE8 application that implements an airline reservation system and runs on top of OpenLiberty. There are two deployments of this application: one that can offload JIT compilations to a Semeru Cloud Compiler service (the one depicted at the bottom) and one that does not benefit from Semeru Cloud Compiler (the one depicted at the top). We will refer to these two deployments as "AcmeAir Baseline" and "AcmeAir with Semeru Cloud Compiler" respectively. Both deployments use the same database (mongodb). Load is applied to both deployments using two identical JMeter containers. Throughput values are sent by the JMeter instances to an influxdb database and from there to a grafana container to be displayed in a dashboard. The two AcmeAir deployments together with the Semeru Cloud Compiler service and mongodb service are deployed in OpenShift and use Knative to scale up and down based on the load. The rest of the pods (for JMeter, influxdb and grafana) run outside of OpenShift on a separate machine.
As part of this lab, the Knative service needs to installed and configured. This will typically be performed by the instructor before the lab and be available for all to use. If you are the instructor, or if you would like to see the steps involved, please refer to the Knative setup instruction.
Login as root using the password provided in the Lab Guide:
su --login root
Clone the repository
cd /home/techzone/Lab-SCC
git clone https://github.com/mpirvu/TechXchange2023.git
cd TechXchange2023
NOTE: If you prefer to use an IDE to view the project, you can run VSCode with admin privileges using the following command:
sudo code --no-sandbox --user-data-dir /home/techzone
Then follow the instructions below.
-
Find the IP of the current machine with
ifconfig. It should be something like10.xxx.xxx.xxxor192.xxx.xxx.x. Then use the./searchReplaceIPAddress.shscript to change all "9.46.81.11" addresses from bash scripts to the IP of the current machine. Use the following command to do this (after replacing xxx.xxx.xxx.xxx with the IP of the current machine)./searchReplaceIPAddress.sh 9.46.81.11 xxx.xxx.xxx.xxx -
Start the influxdb container in normal operation mode:
./startInflux.shVerify the influxdb container is up and running:
podman ps -a | grep influxdbCreate another data bucket:
podman exec influxdb influx bucket create -n jmeter2 -o IBM -r 1dVerify that two buckets called jmeter and jmeter2 exist:
podman exec influxdb influx bucket list -
Create the JMeter container:
cd BuildImages/JMeterContext./build_jmeter.shcd ../.. -
Create the mongodb container
cd BuildImages/MongoContext./build_mongo.shcd ../.. -
Create the two AcmeAir containers (with and without InstantON).
cd BuildImages/LibertyContext./build_acmeair.sh./checkpoint.shcd ../.. -
Push the images for AcmeAir and mongodb to the OCP private repository
IMPORTANT: Login to the OCP console, using the desktop URL provided in the TechZone OCP server reservation.
From the OCP console UI, click the username in the top right corner, and select
Copy login command.Press
Display Tokenand copy theLog in with this tokencommand.Paste the command into your terminal window. You should receive a confirmation message that you are logged in.
Once logged in, create and switch to the
scclab-[Your_initials]namespace:NOTE: If you are working on a cluster that is shared with others, please ensure that you are using a unique project name. We recommend using the format scclab- followed by your initials. For example, scclab-rm.
export CURRENT_NS=scclab-[Your_initials]oc new-project $CURRENT_NS oc project $CURRENT_NSAfter that, log in to the OCP registry:
oc registry login --insecure=trueEnable the default registry route in OCP to push images to its internal repos
oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=mergeLog Podman into the OCP registry server
First we need to get the
TOKENthat we can use to get the password for the registry.oc get secrets -n openshift-image-registry | grep cluster-image-registry-operator-tokenTake note of the
TOKENvalue, as you need to substitute it in the following command that sets the registry password.export OCP_REGISTRY_PASSWORD=$(oc get secret -n openshift-image-registry cluster-image-registry-operator-token-<INPUT_TOKEN> -o=jsonpath='{.data.token}{"\n"}' | base64 -d)Now set the OCP registry host value.
export OCP_REGISTRY_HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}')Finally, we have the values needed for podman to login into the OpenShift registry server.
podman login -p $OCP_REGISTRY_PASSWORD -u kubeadmin $OCP_REGISTRY_HOST --tls-verify=falseNow tag and push the images for AcmeAir and mongodb to the OCP private repository.
- mongo-acmeair image
podman tag localhost/mongo-acmeair:5.0.17 $(oc registry info)/$(oc project -q)/mongo-acmeair:5.0.17 podman push $(oc registry info)/$(oc project -q)/mongo-acmeair:5.0.17 --tls-verify=false
- liberty-acmeair-ee8 image
podman tag localhost/liberty-acmeair-ee8:24.0.0.4 $(oc registry info)/$(oc project -q)/liberty-acmeair-ee8:24.0.0.4 podman push $(oc registry info)/$(oc project -q)/liberty-acmeair-ee8:24.0.0.4 --tls-verify=false
- liberty-acmeair-ee8 instantOn image
podman tag localhost/liberty-acmeair-ee8:24.0.0.4-instanton $(oc registry info)/$(oc project -q)/liberty-acmeair-ee8:24.0.0.4-instanton podman push $(oc registry info)/$(oc project -q)/liberty-acmeair-ee8:24.0.0.4-instanton --tls-verify=false
- Verify the images have been pushed to the OpenShift image repository
oc get imagestream
-
Start the grafana container:
./startGrafana.sh -
Using the grafana UI in a local browser (http://localhost:3000), configure grafana to get data from influxdb.
Note: Initial credentials are admin/admin. You can skip the password change.
Note: There are two pre-configured datasources called InfluxDB and InfluxDB2. However, the IP address of these datasources needs to be adjusted.
To configure the datasources, select the gear (Configuration) from the left side menu, then select "Data sources", then select the data source you want to configure (InfluxDB or InfluxDB2). For the "URL" field of the data source, change the existing IP (9.46.81.11) to the IP machine that runs InfluxDB (this was determined in Step 1 using
ifconfig).Then press "Save & Test" (at the bottom of the screen)to validate the connection. Make sure you change both data sources.
-
Display the pre-loaded dashboard in grafana UI
From the left side menu select the "Dashboard" icon (4 squares), then select "Browse", then select "JMeter Load Test". Activate the automatic refresh feature for the graphs by navigating to the top-right menu of the Grafana dashboard and selecting the
10 secoption, as illustrated in the image below. -
Deploy the services in OCP
-
Update the requisite YAML files with the project namespace you set earlier (CURRENT_NS)
./searchReplaceNs.sh -
Go to the Knative directory:
cd Knative -
Validate that yaml files have the correct images specified:
grep "image:" *.yamlThe image should start with
image-registry.openshift-image-registry.svc:5000/followed by the name of the project where the images were pushed (scclab-[Your_initials]) and followed by the image name and tag. -
Configure mongodb storage by defining a PersistentVolumeClaim with a specific storageClassName
Run the following command to query default storageclass
oc get storageclassSelect the default storageClassName, such as
ocs-storagecluster-xxxxxx (default). Use the storageClassName value (ocs-storagecluster-xxxxxx) replace[Input default storageclass]in the Mongo.yaml file. -
Deploy mongodb:
kubectl apply -f Mongo.yamland validate that the pod is running with:
kubectl get pods | grep mongodb -
Restore the mongo database:
./mongoRestore.sh -
Deploy Semeru Cloud Compiler:
kubectl apply -f JITServer.yamland verify it started sucessfully with:
kubectl get pods | grep jitserver -
Deploy the default AcmeAir instance:
kubectl apply -f AcmeAirKN_default.yamlA message should appear in the console saying that the service was created.
-
Deploy the AcmeAir instance with Semeru Cloud Compiler:
kubectl apply -f AcmeAirKN_SCC.yamlNote: if you want to deploy the AcmeAir instance with Semeru Cloud Compiler and InstantON instead, then follow these steps:
-
To confirm whether the
containerspec-addcapabilitiesis enabled, you can inspect the current configuration ofconfig-featuresby executing the commandkubectl -n knative-serving get cm config-features -oyaml | grep -c "kubernetes.containerspec-addcapabilities: enabled" && echo "true" || echo "false"
IMPORTANT: If the command returns true, it indicates that the Knative 'containerspec-addcapabilities' feature is already enabled. Please skip the step regarding editing Knative permissions. However, if it returns false, please contact your instructor regarding this. In a production scenario, you may be required to enable
containerspec-addcapabilitiesmanually, please refer to our knative setup instruction for further info. -
Create a Service Account bound to your project namespace:
oc create serviceaccount instanton-sa-$CURRENT_NS -
Create a Security Context Constraint named
cap-cr-scc:oc apply -f scc-cap-cr.yaml -
Add the
instanton-saService Account to thecap-cr-sccSecurity Context Constraint:oc adm policy add-scc-to-user cap-cr-scc -z instanton-sa-$CURRENT_NS -
Deploy the AcmeAir instance with Semeru Cloud Compiler and InstantON:
IMPORTANT: Please ensure to fill in all
[Your_initials]fields with the namespace used in the creation step above before proceeding to apply the YAML file.kubectl apply -f AcmeAirKN_SCC_InstantON.yaml
-
-
Verify that 4 pods are running:
kubectl get podsNote: Knative will terminate the AcmeAir pods automatically after about two minutes of inactivity. This does not affect the experiment.
-
-
Apply external load
-
Find the external address of the two AcmeAir services. Use
kubectl get all | grep httpRun the
replaceServerAddress.shscript to automatically identify and replace the[OCP server name]in therunJMeter.shscript:./replaceServerAddress.shAlternatively, you can manually extract the part following
http://orhttps://from the output of the commandkubectl get all | grep http, and then use it as the address for the service in theJHOSTvariable. It should be something likeacmeair-baseline-default.apps.[OCP server name].cloud.techzone.ibm.comandacmeair-scc-default.apps.[OCP server name].cloud.techzone.ibm.com(oracmeair-sccio-default.apps.[OCP server name].cloud.techzone.ibm.comfor the service with InstantON), and replace the example service address in therunJMeter.sh.The same information can be obtained with
knif installed:kn service list -
Verify that the
runJMeter.shscript contains these service addresses for the JHOST environment variable passed to the JMeter containers:IMPORTANT: Please ensure to fill in all fields marked as
[Your_initials]with the namespace used in the creation steps above, and fill in all fields marked as[OCP server name]with the OCP server address you created with TechZone (see the example in the comments mentioned in the "Find the external address of the two AcmeAir services" step) before proceeding to run the runJMeter.sh file.cat runJMeter.sh | grep JHOSTNote: if you selected to start the
AcmeAirKN_SCC_InstantONservice instead ofAcmeAirKN_SCC, then editrunJMeter.shto comment out the second container invocation (the one with JHOST="acmeair-scc-scclab-[Your_initials].apps.[OCP server name].cloud.techzone.ibm.com") and remove the comment from the third container invocation (the one with JHOST="acmeair-sccio-scclab-[Your_initials].apps.[OCP server name].cloud.techzone.ibm.com"). -
Launch jmeter containers:
./runJMeter.shThis will launch two JMeter containers that will generate load for the two AcmeAir services.
After 10 seconds or so, validate that there are no errors with
podman logs jmeter1podman logs jmeter2NOTE: If you encounter any issues while executing this step, please refer to the troubleshooting notes provided.
-
Go to the grafana dashboard you configured before and watch the throughput results for the two services.
The AcmeAirEE8 service with Semeru Cloud Compiler should reach peak throughput much faster than the baseline service.
-
-
Cleanup
- Delete the services you created:
kubectl delete -f AcmeAirKN_default.yaml kubectl delete -f AcmeAirKN_SCC.yaml kubectl delete -f AcmeAirKN_SCC_InstantON.yaml kubectl delete -f JITServer.yaml kubectl delete -f Mongo.yaml - Stop the grafana and influxdb containers:
podman stop grafana podman stop influxdb
- Delete the services you created:
- If there are any errors found in the JMeter service pod logs during the JMeter load test, please note that some of these issues might be transient and could appear due to the heavy loads of traffic. For this exercise, please ignore these transient errors.




