This is a demo to compare reactive with servlet typ of application under load.
The reactive framework used in this demo is Eclipse Vert.x and for the servlet based application we use Spring-Boot with Netflix libraries like Feign and Hystrix.
NOTE: The version of Hystrix used in the demo is actually using Java RX, which is a reactive library, however the servlet container is still using traditional servlet technologies.
NOTE: More modern versions of Spring-Boot also provide a web server implemented using the Reactive system principles called Flux. However, that is still a fairly new and rarely used feature of Spring Boot.
The demo uses Istio and particularly Jaeger tracing to visualise the calls between services.
The purpose of this demo is to show that a reactive system behaves, much better under load then a systems that scales based on threads (e.g. Most servlet engines). This is not to bash Spring-Boot, the same issue will occure in Tomcat and most Java EE implementations as well.
When a servlet engine is put under load and is running out of resources, it will start responding with 503 status code, while using the same load or higher on a reactive system can scale all the way up to the limits of the machine (e.g. CPU, memory, etc).
This demo also shows how Istio and Jaeger tracing can be used to get an greater understanding on how you systems behaves and also to inject failure scenarios.
The installation is used as Ansible Playbook and there for require Ansible and Ansible Galaxy. For load-testing we will use a tool called Siege.
|NOTE: Currently the Ansible playbooks for minishift does only support the CDK versions of Minishift. If you are using community version of minishift then start minishift before you run the playbook and then use -e minishift=false as a parameter to the ansible-playbook.
-
You need to have a OpenShift 3.9 or 3.10 installation, for example Red Hat CDK.
-
Install
ansibleandansible-galaxycommand line toolsMac OSX:
brew install ansible
RHEL 7 or CentOS 7:
sudo yum install -y ansible
-
Install Siege, Siege is an http load testing and benchmarking utility that supports HTTP 1.1 and HTTP 2.0, which is required by Istio.
Mac OSX:
brew install siege
RHEL 7 or CentOS 7:
sudo yum install -y epel-release sudo yum install -y siege
- Clone this repo
git clone https://github.com/tqvarnst/reactive-vs-servlet.git && reactive-vs-servlet
For Minishift use:
-
Run the ansible playbook:
cd ansible ansible-galaxy install -r requirements.yml ansible-playbook init.yml -e openshift_cli=$(which oc)
-
Done!
For standard OpenShift installations:
-
Login via the
occommand line tool as a user withcluster-adminaccess, for example:oc login -u system:admin
-
Run the ansible playbook:
cd ansible ansible-galaxy install -r requirements.yml ansible-playbook init.yml -e minishift=false -e openshift_cli=$(which oc)
-
Done!
I strongly recommend that you find your own way to present this demo, but below is an example story board for running the demo.
-
Open a browser to the OpenShift console and login as user developer.
-
Go into the Coolstore project.
-
Explain the different services in the Coolstore project like this:
- Catalog - A service that returns a list of products that are currently offered on the web site.
- Inventory - A service that returns the inventory status for a product.
- Gateway-XXX - There are two different version of this service that is acting as a microservices gateway and aggregates content from the catalog service and then calls the inventory service to collect the inventory status for each product. One is implemented using Spring-Boot and the other one using Eclipse Vert.x.
- Web - The web project is a NodeJS application that serve as a web server.
- Istio Ingress - The ingress router is what exposes the application to the outside (from OpenShift)
- Istio Route rule - The istio route rules will route traffic from the Gateway service to either the spring-boot pod or the vertx-pod. Default this will be using the Spring Boot route
- Istio Delay route rule - This is a fault injection route where a delay is introduced between the gateway and the inventory service. This rule is used to test how the application behaves when inventory is starting to respond slower.
-
Open a browser to the application using the following command
For Minishift:
minishift openshift service istio-ingressgateway -n istio-system --in-browserFor standard OpenShift installations:
Open the a browser to the result of the following command:
oc get route istio-ingressgateway -n istio-system -o jsonpath="{.spec.host}"Reload the pages a couple of times to show that respond similarly.
-
Show the Istio virtual service pointing to the gateway type-spring
oc get virtualservice gateway -o yaml -
Edit the virtual service and replace type-spring with type-vertx
oc edit virtualservice gateway -
Reload the browser tab pointing to the application and see how the We are Reactive logo in the right upper corner. Reload a couple of more times.
-
Open the Jaeger UI in a browser
For Minishift use:
minishift openshift service tracing -n istio-system --in-browserFor standard OpenShift installations:
Run
oc get route tracing -n istio-system -o jsonpath='{.status.ingress[0].host}{"\n"}'to get the URL -
Select service gateway and find a trace that is lists
-
Select a trace that contains gateway-spring, catalog and inventory
-
Review the call graph and see how request to inventory are called in sequence.
-
Repeat for trace that doesn't contain gateway-spring, but does contain gateway, catalog and inventory and see how the request are called in parallel.
-
Switch back to use the Spring-boot version of the gateway.
oc edit virtualservice gateway -
Introduce a delay to the inventory service
istioctl create -f files/istio/inventory-fault-delay.yml -
Open a terminal and run the following command to start putting load on the Spring gateway.
siege -r 10 -c 10 $(minishift openshift service istio-ingressgateway -n istio-system --url)/api/productsThis should produce a mixture of 200 and 504 and print a summary that looks like this:
Transactions: 756 hits Availability: 94.50 % Elapsed time: 17.98 secs Data transferred: 1.33 MB Response time: 0.12 secs Transaction rate: 42.05 trans/sec Throughput: 0.07 MB/sec Concurrency: 5.12 Successful transactions: 756 Failed transactions: 44 Longest transaction: 5.04 Shortest transaction: 0.02
NOTE: You might see much higher or lower availability depending on how fast your system is. Try increasing or decressing the number of users (E.g.
-c) im the command above if you are not seeing similar results. -
Point out that even though Spring-Boot responds fast, but there are also a high number of failed calls.
-
Switch to the Vert.x version
oc edit virtualservice gateway -
Now, repeat the same test for Eclipse Vert.x using the same command:
siege -r 10 -c 10 $(minishift openshift service istio-ingressgateway -n istio-system --url)/api/productsThis should produce a result with only 200 and print a summary that looks like this:
Transactions: 800 hits Availability: 100.00 % Elapsed time: 20.09 secs Data transferred: 1.50 MB Response time: 0.20 secs Transaction rate: 39.82 trans/sec Throughput: 0.07 MB/sec Concurrency: 8.13 Successful transactions: 800 Failed transactions: 0 Longest transaction: 0.46 Shortest transaction: 0.02
-
Point out that Vert.x has produced a result that has a 100% availability.
-
To summarize, explain that the event queue in Eclipse Vert.x automatically applies the Back-Pressure pattern causing Vert.x to slow down when the system is under pressure rather then return
503status code.
- Mac OS X users need to upgrade the default Python installed to avoid certificate issues with GitHub. See https://blog.bearandgiraffe.com/ansible-github-and-a-failed-to-validate-the-ssl-certificate-story-a8a10ea17512 for more details on how to upgrade the ansible to use a later version of python.
- The Ansible Galaxy roles requires
jmespathpython package on Mac OS X. To install it run:sudo pip3 install jmespath - Ansible on Mac OS X needs the GNU tar instead of BSD tar. Install it via
brew install gnu-tar