Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pod-delete): Adding pod-delete experiment in go #1

Merged
merged 2 commits into from
May 14, 2020
Merged

feat(pod-delete): Adding pod-delete experiment in go #1

merged 2 commits into from
May 14, 2020

Conversation

ispeakc0de
Copy link
Member

@ispeakc0de ispeakc0de commented Apr 22, 2020

Signed-off-by: shubhamchaudhary shubham.chaudhary@mayadata.io

Issue reference: litmuschaos/litmus#1466

@ispeakc0de
Copy link
Member Author

ispeakc0de commented May 12, 2020

Logs of pod-delete job

W0513 14:04:40.832592       1 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0513 14:04:40.833218       1 pod-delete.go:26] [PreReq]: Getting the ENV for the  experiment
I0513 14:04:40.833226       1 pod-delete.go:33] [PreReq]: Updating the chaos result of pod-delete experiment (SOT)
I0513 14:04:40.856016       1 pod-delete.go:42] [Info]: Display the application information passed via the test job
I0513 14:04:40.858286       1 pod-delete.go:43] The application info is as follows: Namespace: test, Label: run=nginx, Ramp Time: 10
I0513 14:04:40.858302       1 pod-delete.go:46] [Status]: Verify that the AUT (Application Under Test) is running (pre-chaos)
I0513 14:04:40.858305       1 application.go:19] [Status]: Checking whether application pods are in running state
I0513 14:04:40.870261       1 application.go:64]  nginx-7989b5f7f4-bssbl Pod is in Running State
I0513 14:04:40.870530       1 application.go:25] [Status]: Checking whether application containers are in running state
I0513 14:04:40.875309       1 application.go:91]  nginx container of pod nginx-7989b5f7f4-bssbl is in Running State
I0513 14:04:40.875404       1 pod-delete.go:72] [Ramp]: Waiting for the 10s ramp time before injecting chaos
I0513 14:04:50.886386       1 pod-delete.go:26] Killing [nginx-7989b5f7f4-bssbl] pods
I0513 14:04:50.912252       1 pod-delete.go:50] [Wait]: Wait for the chaos interval 10s
I0513 14:05:00.912727       1 pod-delete.go:54] [Status]: Verification for the recreation of application pod
I0513 14:05:00.912770       1 application.go:19] [Status]: Checking whether application pods are in running state
I0513 14:05:00.925912       1 application.go:64]  nginx-7989b5f7f4-n6jnt Pod is in Running State
I0513 14:05:00.926112       1 application.go:25] [Status]: Checking whether application containers are in running state
I0513 14:05:00.936866       1 application.go:91]  nginx container of pod nginx-7989b5f7f4-n6jnt is in Running State
I0513 14:05:00.942681       1 pod-delete.go:26] Killing [nginx-7989b5f7f4-n6jnt] pods
I0513 14:05:00.958139       1 pod-delete.go:50] [Wait]: Wait for the chaos interval 10s
I0513 14:05:10.958581       1 pod-delete.go:54] [Status]: Verification for the recreation of application pod
I0513 14:05:10.958773       1 application.go:19] [Status]: Checking whether application pods are in running state
I0513 14:05:10.969005       1 application.go:64]  nginx-7989b5f7f4-n28kf Pod is in Running State
I0513 14:05:10.969135       1 application.go:25] [Status]: Checking whether application containers are in running state
I0513 14:05:10.984484       1 application.go:91]  nginx container of pod nginx-7989b5f7f4-n28kf is in Running State
I0513 14:05:10.988964       1 pod-delete.go:26] Killing [nginx-7989b5f7f4-n28kf] pods
I0513 14:05:11.007312       1 pod-delete.go:50] [Wait]: Wait for the chaos interval 10s
I0513 14:05:21.008002       1 pod-delete.go:54] [Status]: Verification for the recreation of application pod
I0513 14:05:21.008110       1 application.go:19] [Status]: Checking whether application pods are in running state
I0513 14:05:23.027403       1 application.go:64]  nginx-7989b5f7f4-tpr9n Pod is in Running State
I0513 14:05:23.027547       1 application.go:25] [Status]: Checking whether application containers are in running state
I0513 14:05:23.039463       1 application.go:91]  nginx container of pod nginx-7989b5f7f4-tpr9n is in Running State
I0513 14:05:23.039735       1 pod-delete.go:82] [Ramp]: Waiting for the 10s ramp time after injecting chaos
I0513 14:05:33.040326       1 pod-delete.go:61] [Confirmation]: The application pod has been deleted successfully
I0513 14:05:33.040771       1 pod-delete.go:70] [Status]: Verify that the AUT (Application Under Test) is running (post-chaos)
I0513 14:05:33.040948       1 application.go:19] [Status]: Checking whether application pods are in running state
I0513 14:05:33.060141       1 application.go:64]  nginx-7989b5f7f4-tpr9n Pod is in Running State
I0513 14:05:33.060465       1 application.go:25] [Status]: Checking whether application containers are in running state
I0513 14:05:33.070032       1 application.go:91]  nginx container of pod nginx-7989b5f7f4-tpr9n is in Running State
I0513 14:05:33.070094       1 pod-delete.go:79] [The End]: Updating the chaos result of pod-delete experiment (EOT)


@ispeakc0de ispeakc0de removed the WIP label May 12, 2020
@ispeakc0de ispeakc0de linked an issue May 12, 2020 that may be closed by this pull request
Signed-off-by: shubhamchaudhary <shubham.chaudhary@mayadata.io>
Copy link
Member

@chandankumar4 chandankumar4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some structuring need to change a bit, Please try to go through this: https://github.com/litmuschaos/litmus/blob/cfd91c26ca1ca668a40d2bc85fdbc23f576de8df/contribute/developer_guide/STYLE_GUIDE.md and change it accordingly

build/litmus-go/Dockerfile Outdated Show resolved Hide resolved
chaoslib/litmus/podDeleteChaos.go Outdated Show resolved Hide resolved
pkg/utils/chaosResult.go Outdated Show resolved Hide resolved
pkg/utils/checkApplicationStatus.go Outdated Show resolved Hide resolved
pkg/utils/getENV.go Outdated Show resolved Hide resolved
pkg/utils/types.go Outdated Show resolved Hide resolved
Comment on lines 40 to 93
func (clientSets *ClientSets) GenerateClientSetFromKubeConfig() error {

config, err := getKubeConfig()
if err != nil {
return err
}
k8sClientSet, err := GenerateK8sClientSet(config)
if err != nil {
return err
}
litmusClientSet, err := GenerateLitmusClientSet(config)
if err != nil {
return err
}
clientSets.KubeClient = k8sClientSet
clientSets.LitmusClient = litmusClientSet
return nil
}

// getKubeConfig setup the config for access cluster resource
func getKubeConfig() (*rest.Config, error) {
kubeconfig := flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
flag.Parse()
// Use in-cluster config if kubeconfig path is specified
if *kubeconfig == "" {
config, err := rest.InClusterConfig()
if err != nil {
return config, err
}
}
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
if err != nil {
return config, err
}
return config, err
}

// GenerateK8sClientSet will generation k8s client
func GenerateK8sClientSet(config *rest.Config) (*kubernetes.Clientset, error) {
k8sClientSet, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, errors.Wrapf(err, "Unable to generate kubernetes clientSet %s: ", err)
}
return k8sClientSet, nil
}

// GenerateLitmusClientSet will generate a LitmusClient
func GenerateLitmusClientSet(config *rest.Config) (*chaosClient.LitmuschaosV1alpha1Client, error) {
litmusClientSet, err := chaosClient.NewForConfig(config)
if err != nil {
return nil, errors.Wrapf(err, "Unable to create LitmusClientSet: %v", err)
}
return litmusClientSet, nil
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move this function in a new file called: pkg/clientset.go

chaoslib/litmus/podDeleteChaos.go Outdated Show resolved Hide resolved
experiments/generic/pod_delete/pod_delete.go Outdated Show resolved Hide resolved
experiments/generic/pod_delete/pod_delete_k8s_job.yml Outdated Show resolved Hide resolved
hack/push Outdated Show resolved Hide resolved
hack/generate_go_binary Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@@ -0,0 +1,22 @@
FROM ubuntu:bionic
Copy link
Member

@ksatchit ksatchit May 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can use a lighter image as well (post 1.4)

)

func main() {
var err error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

##########################
vars: --
##########################

experimentsDetails := utils.ExperimentDetails{}
clients := utils.ClientSets{}

//Getting kubeConfig and Generate ClientSets
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

########################
environment: - 
########################

if err != nil {
klog.V(0).Infof("Unable to Create the Chaos Result due to %v\n", err)
failStep = "Updating the chaos result of pod-delete experiment (SOT)"
utils.UpdateChaosResultMiddle(&experimentsDetails, clients, flag,failStep); return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

}

//PRE-CHAOS AUXILIARY APPLICATION LIVENESS CHECK
if experimentsDetails.AuxiliaryAppInfo == "no" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we update this?

err = litmus.PreparePodDelete(&experimentsDetails, clients,&failStep)
if err != nil {
klog.V(0).Infof("Chaos injection failed due to %v\n", err)
utils.UpdateChaosResultMiddle(&experimentsDetails, clients, flag,failStep); return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name for result fn


//Updating the chaosResult in the end of experiment
klog.V(0).Infof("[The End]: Updating the chaos result of %v experiment (EOT)",experimentsDetails.ExperimentName)
err = utils.UpdateChaosResultEnd(&experimentsDetails, clients, flag,failStep)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fn name

@@ -0,0 +1,93 @@
package utils

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can have single fn taking action and values (post 1.4.0)

// UpdateChaosResultMiddle Update the chaos result (Middle of Test)
func UpdateChaosResultMiddle(experimentsDetails *ExperimentDetails, clients ClientSets, flag string, failStep string) {

klog.V(0).Infof("[The End]: Updating the chaos result of %v experiment (EOT)",experimentsDetails.ExperimentName)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Log optimization -- we might not lead to log every action that we are performing -- only user relevant info is sufficient.

Signed-off-by: shubhamchaudhary <shubham.chaudhary@mayadata.io>
Copy link
Member

@ksatchit ksatchit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Copy link
Member

@chandankumar4 chandankumar4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more improvement is required based on the import path and function rearrangement, We can improve in the upcoming PRs
/LGTM
Thanks @ispeakc0de

Copy link
Member

@rahulchheda rahulchheda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/LGTM
Added some small comments, nothing major, we can do post 1.4.x
Thanks @ispeakc0de , great PR!

Makefile Outdated
@echo "------------------"
@echo "--> Build litmus-go image"
@echo "------------------"
sudo docker build . -f build/litmus-go/Dockerfile -t shubh214/litmus-go:ci2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker repo name change needed

if err != nil {
return err
}
klog.V(0).Infof("Killing %v pods", targetPodList)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
klog.V(0).Infof("Killing %v pods", targetPodList)
klog.Infof("Killing %v pods", targetPodList)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like you are not using other Verbosity Levels, so you might not need to set it also.

labels:
experiment: pod-delete
spec:
# Placeholder that is updated by the executor for automated runs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unable to get the context of this comment

serviceAccountName: litmus
restartPolicy: Never
containers:
- name: gotest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use appropriate container name? something like go-exp

golang.org/x/sys v0.0.0-20200509044756-6aff5f38e54f // indirect
golang.org/x/time v0.0.0-20200416051211-89c76fbcd5d1 // indirect
google.golang.org/appengine v1.6.6 // indirect
k8s.io/apimachinery v0.0.0-20190817020851-f2f3a405f61d
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a pin comment here to help others understand this timestamp

// CheckPodStatus checks the status of the application pod
func CheckPodStatus(appNs string, appLabel string, clients environment.ClientSets) error {
err := retry.
Times(90).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if got running before 180sec ? are we handling that in this retry func?

@ksatchit ksatchit merged commit 80fb308 into litmuschaos:master May 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide go-library for scaffolding of expeirments
4 participants