6fault-injection.adoc

Chaos Testing

Apply some chaos engineering by throwing in some HTTP errors or network delays. Understanding failure scenarios is a critical aspect of microservices architecture (aka distributed computing)

Important

Before Start You should have NO virtualservice nor destinationrule (in tutorial namespace) kubectl get virtualservice{namespace-suffix} kubectl get destinationrule{namespace-suffix} if so run: ./scripts/clean.sh tutorial{namespace-suffix}

HTTP Error 503

By default, recommendation v1 and v2 are being randomly load-balanced as that is the default behavior in Kubernetes/OpenShift

$ oc get pods -l app=recommendation -n tutorial{namespace-suffix}
or
$ kubectl get pods -l app=recommendation -n tutorial{namespace-suffix}

NAME                                  READY     STATUS    RESTARTS   AGE
recommendation-v1-3719512284-7mlzw   2/2       Running   6          18h
recommendation-v2-2815683430-vn77w   2/2       Running   0          3h

You can inject 503’s, for approximately 50% of the requests

kubectl create -f istiofiles/destination-rule-recommendation.yml -n tutorial{namespace-suffix}
kubectl create -f istiofiles/virtual-service-recommendation-503.yml -n tutorial{namespace-suffix}

./scripts/run.sh http://istio-ingressgateway-istio-system.{appdomain}/{path}

customer => preference => recommendation v2 from '7778d6fb89-j6lts': 138
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v1 from '6976858b48-dswhz': 139
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 139
customer => preference => recommendation v1 from '6976858b48-dswhz': 140
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 140
customer => preference => recommendation v1 from '6976858b48-dswhz': 141
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => Error: 503 - preference => Error: 503 - fault filter abort
customer => preference => recommendation v2 from '7778d6fb89-j6lts': 141

Clean up

kubectl delete -f istiofiles/virtual-service-recommendation-503.yml -n tutorial{namespace-suffix}

Delay

The most insidious of possible distributed computing faults is not a "down" service but a service that is responding slowly, potentially causing a cascading failure in your network of services.

kubectl create -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial{namespace-suffix}
kubectl replace -f istiofiles/destination-rule-recommendation.yml -n tutorial{namespace-suffix}

And hit the customer endpoint

./scripts/run.sh http://istio-ingressgateway-istio-system.{appdomain}/{path}

You will notice many requests to the customer endpoint now have a delay. If you are monitoring the logs for recommendation v1 and v2, you will also see the delay happens BEFORE the recommendation service is actually called

stern recommendation -n tutorial{namespace-suffix}

or `bash ./kubetail.sh recommendation -n tutorial{namespace-suffix} `

Clean up

kubectl delete -f istiofiles/destination-rule-recommendation.yml -n tutorial{namespace-suffix}
kubectl delete -f istiofiles/virtual-service-recommendation-delay.yml -n tutorial{namespace-suffix}