Aws az down

[damienomurchu]

What

Add litmus experiment to simulate failure of an aws az.

Why

Evaluate resiliency of application under test if an az goes down.

How

• Add new litmus-go experiment for az-down
• create dummy ACLs with deny all traffic policies
• assign ACLs to subnets of az under test
• revert ACL changes after experiment concludes

[damienomurchu]

Thanks for the eyes @ksatchit, I'm having a little trouble verifying it using okteto which I documented in #229 (before I created this PR) if I'm missing anything besides the serviceaccount and the litmus-experiment I should deploy to test locally

[ksatchit]

Hi @damienomurchu ! once again, thanks for working on the experiment. Have shared some info here regarding the tests. At this point, okteto is more for repeated devtest. The tests w/ chaosexperiment and chaosengine is a separate step - not integrated w/ okteto as such.

[damienomurchu]

I've been running this locally with okteto and verified all but the destructive aspects of the experiment involving the reassignment of the blocking ACL's to take the targeted az instances out of circulation.

The multi-az cluster I've been testing this against is due to expire, so I will test this on a new cluster on Monday most likely. All going well I'll tidy up the PR and take it out of draft.

Thanks for the helpful info and fix on #229 !

[ksatchit]

Thanks @damienomurchu !! Also Adding other community members specifically interested in this experiment for their perspective/reviews - cc: @kazukousen @suhrud-kumar

[damienomurchu]

Most of the az-down business logic is in the experiment right now, but will look to refactor and move here

[damienomurchu]

Mostly verified by running experiment locally with okteto against a multi-az cluster Will run against a fresh cluster later today by regular litmus workflow.

[damienomurchu]

Hmm, getting a panic over a nil map assignment I wasn't getting when running the experiment locally with okteto

❯ oc logs -f pod/az-down-chaos-runner                                    
W1215 15:51:49.797393       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2020-12-15T15:51:49Z" level=info msg="Experiments details are as follows" Experiments List="[az-down]" Engine Name=az-down-chaos appLabels="app.kubernetes.io/name=kafka" appKind= Service Account Name=az-down-sa Engine Namespace=litmus
panic: assignment to entry in nil map

goroutine 1 [running]:
github.com/litmuschaos/chaos-runner/pkg/utils.(*ExperimentDetails).SetLabels(...)
	/home/circleci/go/src/github.com/litmuschaos/chaos-runner/pkg/utils/experimentHelpers.go:168
github.com/litmuschaos/chaos-runner/pkg/utils.(*ExperimentDetails).SetValueFromChaosExperiment(0xc420891ba8, 0xc4201c3a40, 0xc4202da100, 0xc420891a00, 0x1, 0x1)
	/home/circleci/go/src/github.com/litmuschaos/chaos-runner/pkg/utils/experimentHelpers.go:40 +0x1d5
github.com/litmuschaos/chaos-runner/pkg/utils.(*ExperimentDetails).SetValueFromChaosResources(0xc420891ba8, 0xc420891a00, 0xc4201c3a40, 0xc4202da100, 0xc4202da1e0, 0x1)
	/home/circleci/go/src/github.com/litmuschaos/chaos-runner/pkg/utils/experimentHelpers.go:206 +0x396
main.main()
	/home/circleci/go/src/github.com/litmuschaos/chaos-runner/bin/runner.go:56 +0x779

I've rebuilt the binary and the image from the latest PR changes before testing. Any maps I'm using seem to be initialised but I seem to be missing something so any fresh eyes would be welcome :)

[damienomurchu]

@ksatchit I feel the PR is probably sufficiently advanced to consider taking it out of draft, but after today I'm mostly AFK for about a month so I will be guided by you as to whether I should take this out of draft before then and resolve any outstanding issues thrown out by the CI.

[ksatchit]

Hi @damienomurchu , yes I guess we can move it out of draft/to actual PR!! Will take a closer look shortly (just wrapping up the 1.11.0 release)

[ksatchit]

Tagging @uditgaurav on the e2e failure on container-kill experiment.

[damienomurchu]

Closing this as we went a different direction on this instead, re-using a shell script using the aws-cli to perform the same steps as in the logic in this experiment. I will leave the PR branch up in case any of this work is useful to others.

Thanks again for the help and feedback with this PR @ksatchit