Eclipse Vert.x Istio Circuit Breaker mission

Purpose

Showcase Istio’s Circuit Breaker via a (minimally) instrumented Eclipse Vert.x microservices.

Prerequisites

• Openshift 3.10 cluster with Istio. For local development, download the latest release from [Maistra](https://github.com/Maistra/origin/releases) and run:

# Set oc to be the Maistra one
oc cluster up --enable="*,istio"
oc login -u system:admin
oc apply -f https://raw.githubusercontent.com/Maistra/openshift-ansible/maistra-0.2.0-ocp-3.1.0-istio-1.0.2/istio/cr-minimal.yaml -n istio-operator
oc get pods -n istio-system -w

Wait until the openshift-ansible-istio-installer-job-xxxx job has completed. It can take several minutes. The OpenShift console is available on https://127.0.0.1:8443.

• Create a new project/namespace on the cluster. This is where your application will be deployed.

oc login -u system:admin
oc adm policy add-cluster-role-to-user admin developer --as=system:admin
oc login -u developer -p developer
oc new-project <whatever valid project name you want> # not required

Build and deploy the application with the Fabric8 Maven Plugin

Execute the following command to build the project and deploy it to OpenShift:

mvn clean fabric8:deploy -Popenshift

Configuration for FMP may be found both in pom.xml and src/main/fabric8 files/folders.

Build and deploy the application with the S2I

find . | grep openshiftio | grep application | xargs -n 1 oc apply -f

oc new-app --template=vertx-istio-circuit-breaker-greeting -p SOURCE_REPOSITORY_URL=https://github.com/openshiftio-vertx-boosters/vertx-istio-circuit-breaker-booster  -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=greeting-service
oc new-app --template=vertx-istio-circuit-breaker-name -p SOURCE_REPOSITORY_URL=https://github.com/openshiftio-vertx-boosters/vertx-istio-circuit-breaker-booster  -p SOURCE_REPOSITORY_REF=master -p SOURCE_REPOSITORY_DIR=name-service

Use Cases

Access the application

• Run the following command to determine the appropriate URL to access our demo. Make sure you access the URL with the HTTP scheme. HTTPS is NOT enabled by default:

  oc create -f rules/gateway.yaml
  echo "http://$(oc get route istio-ingressgateway -o jsonpath='{.spec.host}{"\n"}' -n istio-system)/breaker/greeting"

Open the given URL.

Verify application behavior

1. Click on the "Start" button to issue 10 concurrent requests to the name service.

2. Click on the "Stop" button to stop the requests

3. You can change the number of concurrent requests between 1 and 20.

4. All calls go through as expected.

Initial Circuit Breaker configuration

1. Now apply the initial DestinationRule that activates Istio’s Circuit Breaker on the name service, configuring it to allow a maximum of 100 concurrent connections.

oc create -f rules/initial-destination-rule.yaml -n $(oc project -q)

1. You can check that the DestinationRule is properly applied by executing:

oc get destinationrules.networking.istio.io

1. Try the application again. Since we only make up to 20 concurrent connections, the circuit breaker should not trip.

Restrictive Circuit Breaker configuration

1. Now apply a more restrictive DestinationRule, after having remove the initial one:

oc delete -f rules/initial-destination-rule.yaml
oc create -f rules/restrictive-destination-rule.yaml

1. Try the application again

Since the Circuit Breaker is now configured to only allow one concurrent connection and by default we are sending 10 to the name service, we should now see the Circuit Breaker tripping open. However, experimentally, we observe that this does not happen in a clear-cut fashion: the circuit is not always open. In fact, depending on how fast the server on which the application is running, the circuit might not break open at all. The reason for this is that is the name service does not do much and thus responds quite fast. This, in turn, leads to not having much concurrency at all.

Optional: fault injection

We could try to increase contention on the name service using Istio’s fault injection behavior by applying a 1-second delay to 50% of the calls to the name service.

1. Create a new VirtualService for the name service to inject a processing delay:

oc create -f rules/name-with-delay.yaml

1. Try the application again

You should observe that this does not seem to change how often the circuit breaks open. This is due to the fact that the injected delay actually occurs between the services. So, in essence, this only time shifts the requests, only increasing concurrency marginally (due to the fact that only 50% of the requests are delayed). This still does not let us observe the circuit breaking open properly.

1. For more comfort, let’s return to the original configuration by deleting the VirtualService we just introduced:

oc delete -f rules/name-with-delay.yaml

Simulate load on the name service

We need to increase contention on the name service in order to have enough concurrent connections to trip open the circuit breaker. We can accomplish this by simulating load on the name service by asking it to introduce a random processing time. To accomplish this:

1. Stop the requests (if that wasn’t already the case)

2. Checking the "Simulate load" checkbox

3. Start the requests.

You should now observe the circuit breaking open by observing lots of Hello, Fallback! messages.

Undeploy the application

With Fabric8 Maven Plugin (FMP)

mvn fabric8:undeploy

Remove the namespace

This will delete the project from the OpenShift cluster.

oc delete project <your project name>