refactor(kafka-broker-pod-failure): Refactor the kafka broker pod failure experiment #309
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ispeakc0de
force-pushed
the
kafka-experiment=refactor
branch
from
16a5b87
to
eb2a4e1
Compare
time="2021-03-08T19:01:53Z" level=info msg="Experiment Name: kafka-broker-pod-failure"
time="2021-03-08T19:01:53Z" level=info msg="[PreReq]: Getting the ENV for the kafka-broker-pod-failure"
time="2021-03-08T19:01:53Z" level=info msg="[PreReq]: Updating the chaos result of kafka-broker-pod-failure experiment (SOT)"
time="2021-03-08T19:01:53Z" level=info msg="The application informations are as follows" Kafka Namespace=kafka Kafka Label="app=kafka" Ramp Time=0
time="2021-03-08T19:01:53Z" level=info msg="[Status]: Verify that the Kafka cluster is healthy(pre-chaos)"
time="2021-03-08T19:01:53Z" level=info msg="[Status]: Verify that all the kafka pods are running"
time="2021-03-08T19:01:53Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:01:53Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-0 Readiness=true container=k8skafka
time="2021-03-08T19:01:53Z" level=info msg="[Status]: The Container status are as follows" container=kafka-node-exporter Pod=kafka-kafka-0 Readiness=true
time="2021-03-08T19:01:53Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-1 Readiness=true
time="2021-03-08T19:01:53Z" level=info msg="[Status]: The Container status are as follows" container=kafka-node-exporter Pod=kafka-kafka-1 Readiness=true
time="2021-03-08T19:01:53Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-2 Readiness=true
time="2021-03-08T19:01:53Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-2 Readiness=true container=kafka-node-exporter
time="2021-03-08T19:01:55Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:01:55Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-0 Status=Running
time="2021-03-08T19:01:55Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-1 Status=Running
time="2021-03-08T19:01:55Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-2 Status=Running
time="2021-03-08T19:01:57Z" level=info msg="[Status]: Verify that all the zookeeper pods are running"
time="2021-03-08T19:01:57Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:01:57Z" level=info msg="[Status]: The Container status are as follows" container=kubernetes-zookeeper Pod=zookeeper-instance-zookeeper-0 Readiness=true
time="2021-03-08T19:01:57Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kubernetes-zookeeper Pod=zookeeper-instance-zookeeper-1
time="2021-03-08T19:01:57Z" level=info msg="[Status]: The Container status are as follows" Pod=zookeeper-instance-zookeeper-2 Readiness=true container=kubernetes-zookeeper
time="2021-03-08T19:01:59Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:01:59Z" level=info msg="[Status]: The status of Pods are as follows" Pod=zookeeper-instance-zookeeper-0 Status=Running
time="2021-03-08T19:01:59Z" level=info msg="[Status]: The status of Pods are as follows" Status=Running Pod=zookeeper-instance-zookeeper-1
time="2021-03-08T19:01:59Z" level=info msg="[Status]: The status of Pods are as follows" Pod=zookeeper-instance-zookeeper-2 Status=Running
time="2021-03-08T19:02:01Z" level=info msg="[Liveness]: Set the kafka topic name"
time="2021-03-08T19:02:01Z" level=info msg="[Liveness]: Creating the kafka liveness pod"
time="2021-03-08T19:02:01Z" level=info msg="[Liveness]: Confirm that the kafka liveness pod is running"
time="2021-03-08T19:02:01Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:02:03Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:02:11Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-liveness-bpuspt Status=Running
time="2021-03-08T19:02:13Z" level=info msg="[Liveness]: Obtain the leader broker ordinality for the topic (partition) created by kafka-liveness"
time="2021-03-08T19:02:15Z" level=info msg="[Liveness]: Determine the leader broker pod name"
time="2021-03-08T19:02:15Z" level=info msg="The Liveness pod gets established"
time="2021-03-08T19:02:15Z" level=info msg="[Info]: Kafka partition leader is kafka-kafka-0"
time="2021-03-08T19:02:15Z" level=info msg="[Info]: Kafka broker pod for deletion is kafka-kafka-0"
time="2021-03-08T19:02:15Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2021-03-08T19:02:15Z" level=info msg="[Info]: Killing the following pods" PodName=kafka-kafka-0
time="2021-03-08T19:02:15Z" level=info msg="[Wait]: Wait for the chaos interval 20s"
time="2021-03-08T19:02:35Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2021-03-08T19:02:35Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:02:59Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-0 Readiness=true
time="2021-03-08T19:02:59Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kafka-node-exporter Pod=kafka-kafka-0
time="2021-03-08T19:02:59Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-1 Readiness=true container=k8skafka
time="2021-03-08T19:02:59Z" level=info msg="[Status]: The Container status are as follows" container=kafka-node-exporter Pod=kafka-kafka-1 Readiness=true
time="2021-03-08T19:02:59Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-2 Readiness=true container=k8skafka
time="2021-03-08T19:02:59Z" level=info msg="[Status]: The Container status are as follows" container=kafka-node-exporter Pod=kafka-kafka-2 Readiness=true
time="2021-03-08T19:03:01Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:03:01Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-0 Status=Running
time="2021-03-08T19:03:01Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-1 Status=Running
time="2021-03-08T19:03:01Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-2 Status=Running
time="2021-03-08T19:03:03Z" level=info msg="[Chaos]:Number of pods targeted: 1"
time="2021-03-08T19:03:03Z" level=info msg="[Info]: Killing the following pods" PodName=kafka-kafka-0
time="2021-03-08T19:03:03Z" level=info msg="[Wait]: Wait for the chaos interval 20s"
time="2021-03-08T19:03:23Z" level=info msg="[Status]: Verification for the recreation of application pod"
time="2021-03-08T19:03:23Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:03:41Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-0 Readiness=true
time="2021-03-08T19:03:41Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-0 Readiness=true container=kafka-node-exporter
time="2021-03-08T19:03:41Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-1 Readiness=true
time="2021-03-08T19:03:41Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kafka-node-exporter Pod=kafka-kafka-1
time="2021-03-08T19:03:41Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-2 Readiness=true container=k8skafka
time="2021-03-08T19:03:41Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kafka-node-exporter Pod=kafka-kafka-2
time="2021-03-08T19:03:43Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:03:43Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-0 Status=Running
time="2021-03-08T19:03:43Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-1 Status=Running
time="2021-03-08T19:03:43Z" level=info msg="[Status]: The status of Pods are as follows" Status=Running Pod=kafka-kafka-2
time="2021-03-08T19:03:45Z" level=info msg="[Chaos]: Time is up for experiment: "
time="2021-03-08T19:03:45Z" level=info msg="[Completion]: chaos is done"
time="2021-03-08T19:03:45Z" level=info msg="[Confirmation]: The application pod has been deleted successfully"
time="2021-03-08T19:03:45Z" level=info msg="[Status]: Verify that the Kafka cluster is healthy(post-chaos)"
time="2021-03-08T19:03:45Z" level=info msg="[Status]: Verify that all the kafka pods are running"
time="2021-03-08T19:03:45Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:03:45Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-0 Readiness=true container=k8skafka
time="2021-03-08T19:03:45Z" level=info msg="[Status]: The Container status are as follows" Pod=kafka-kafka-0 Readiness=true container=kafka-node-exporter
time="2021-03-08T19:03:45Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-1 Readiness=true
time="2021-03-08T19:03:45Z" level=info msg="[Status]: The Container status are as follows" container=kafka-node-exporter Pod=kafka-kafka-1 Readiness=true
time="2021-03-08T19:03:45Z" level=info msg="[Status]: The Container status are as follows" container=k8skafka Pod=kafka-kafka-2 Readiness=true
time="2021-03-08T19:03:45Z" level=info msg="[Status]: The Container status are as follows" container=kafka-node-exporter Pod=kafka-kafka-2 Readiness=true
time="2021-03-08T19:03:47Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:03:47Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-0 Status=Running
time="2021-03-08T19:03:47Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-1 Status=Running
time="2021-03-08T19:03:47Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-kafka-2 Status=Running
time="2021-03-08T19:03:49Z" level=info msg="[Status]: Verify that all the zookeeper pods are running"
time="2021-03-08T19:03:49Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:03:49Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kubernetes-zookeeper Pod=zookeeper-instance-zookeeper-0
time="2021-03-08T19:03:49Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kubernetes-zookeeper Pod=zookeeper-instance-zookeeper-1
time="2021-03-08T19:03:49Z" level=info msg="[Status]: The Container status are as follows" container=kubernetes-zookeeper Pod=zookeeper-instance-zookeeper-2 Readiness=true
time="2021-03-08T19:03:51Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:03:51Z" level=info msg="[Status]: The status of Pods are as follows" Pod=zookeeper-instance-zookeeper-0 Status=Running
time="2021-03-08T19:03:51Z" level=info msg="[Status]: The status of Pods are as follows" Pod=zookeeper-instance-zookeeper-1 Status=Running
time="2021-03-08T19:03:51Z" level=info msg="[Status]: The status of Pods are as follows" Status=Running Pod=zookeeper-instance-zookeeper-2
time="2021-03-08T19:03:53Z" level=info msg="[Status]: Verify that the Kafka liveness pod is running(post-chaos)"
time="2021-03-08T19:03:53Z" level=info msg="[Status]: Checking whether application containers are in ready state"
time="2021-03-08T19:03:53Z" level=info msg="[Status]: The Container status are as follows" container=kafka-consumer Pod=kafka-liveness-bpuspt Readiness=true
time="2021-03-08T19:03:53Z" level=info msg="[Status]: The Container status are as follows" Readiness=true container=kafka-producer Pod=kafka-liveness-bpuspt
time="2021-03-08T19:03:55Z" level=info msg="[Status]: Checking whether application pods are in running state"
time="2021-03-08T19:03:55Z" level=info msg="[Status]: The status of Pods are as follows" Pod=kafka-liveness-bpuspt Status=Running
time="2021-03-08T19:03:57Z" level=info msg="[CleanUp]: Deleting the kafka liveness pod(post-chaos)"
time="2021-03-08T19:04:37Z" level=info msg="[The End]: Updating the chaos result of kafka pod delete experiment (EOT)"
|
…lure Signed-off-by: shubhamchaudhary <shubham@chaosnative.com>
ispeakc0de
force-pushed
the
kafka-experiment=refactor
branch
from
eb2a4e1
to
52e3f78
Compare
ksatchit
approved these changes
Mar 12, 2021
uditgaurav
approved these changes
Mar 15, 2021
ispeakc0de
added a commit
to ispeakc0de/litmus-go
that referenced
this pull request
Mar 15, 2021
…lure (litmuschaos#309) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com>
ksatchit
pushed a commit
that referenced
this pull request
Mar 15, 2021
* chore(aut-check): skip autStatus check if appinfo not provided (#304) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com> * chore(n/w-chaos): Handling the unknown hosts (#302) Signed-off-by: shubhamchaudhary <shubham.chaudhary@mayadata.io> * chore(signal): Adding signal for the crio/containerd runtime (#306) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com> * chore(disk-fill): Addding option to specify ephemeral storage(Mibi) explictly via env (#313) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com> * refactor(kafka-broker-pod-failure): Refactor the kafka broker pod failure (#309) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com> * Remove init container for changing permission of the container runtime socket (#315) * Removed init container from container-kill and network-chaos experiments Signed-off-by: Radu Domnu <rdomnu@redhat.com> * Running crictl/docker commands with sudo Signed-off-by: Radu Domnu <rdomnu@redhat.com> * Removed init container for network experiments Signed-off-by: Radu Domnu <rdomnu@redhat.com> * chore(k8sProbe): Updating the k8s probe schema (#308) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com> * api response time updated for http probe (#307) Signed-off-by: oumkale <oum.kale@mayadata.io> * chore(container-status): checking only target container status (#303) Signed-off-by: shubhamchaudhary <shubham@chaosnative.com> * enhance(aws-ec2): Add support for terminating node of a cluster with self-managed nodegroup (#298) * enhanc(aws-ec2): Add support for terminating node of a cluster with self-managed nodegroup Signed-off-by: udit <udit@chaosnative.com> Co-authored-by: Radu Domnu <39598837+radudd@users.noreply.github.com> Co-authored-by: OUM NIVRATHI KALE <oum.kale@mayadata.io> Co-authored-by: Udit Gaurav <35391335+uditgaurav@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: shubhamchaudhary shubham@chaosnative.com
Refactor the Kafka broker pod failure experiment
Tested for