Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug Report in Kubelet-service-kill experiment #1629

Closed
Shreyangi opened this issue Jul 1, 2020 · 2 comments
Closed

Bug Report in Kubelet-service-kill experiment #1629

Shreyangi opened this issue Jul 1, 2020 · 2 comments
Assignees
@uditgaurav
Labels
bug kind/bug
Milestone
1.5.1

Comments

@Shreyangi
Copy link

In the kubelet-service-kill experiment, error converting in YAML to JSON

What happened:

kubectl logs -f kubelet-service-kill-rndfdt-sgvqw -n default
ansible-playbook 2.7.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.17 (default, Apr 15 2020, 17:20:14) [GCC 7.5.0]
Using /etc/ansible/ansible.cfg as config file
/etc/ansible/hosts did not meet host_list requirements, check plugin documentation if this is unexpected
/etc/ansible/hosts did not meet script requirements, check plugin documentation if this is unexpected
statically imported: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_prerequisites.yml

2020-06-30 14:55:20.251221 PLAYBOOK: kubelet_service_kill_ansible_logic.yml 
1 plays in ./experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml

2020-06-30 14:55:20.271076 ************ BRACE YOURSELF, EXPERIMENT BEGINS! ************ 

2020-06-30 14:55:23.559301 Step: Gathering Facts 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:2
META: ran handlers

2020-06-30 14:55:24.123067 Step: [PreReq] Identify the chaos util for kubelet-service-kill experiment 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_prerequisites.yml:1
details => {"changed": true, "checksum": "2d76d919fa3c20d76eff0b8e3148897f4319c491", "dest": "/tmp/chaosutil.yml", "gid": 0, "group": "root", "md5sum": "c855258e0f920687b5ce930bafa965a7", "mode": "0644", "owner": "root", "size": 73, "src": "/tmp/ansible-tmp-1593528923.62-232749329164298/source", "state": "file", "uid": 0}

2020-06-30 14:55:24.187852 Step: [PreReq]: Including the chaos util for the kubelet-service-kill experiment 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:27
details => {"ansible_facts": {"c_util": "/chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml"}, "ansible_included_var_files": ["/tmp/chaosutil.yml"], "changed": false}

2020-06-30 14:55:24.253656 Step: [PreReq]: Constructing the chaos result name 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:32
details => {"ansible_facts": {"c_result": "nginx-chaos-kubelet-service-kill"}, "changed": false}
included: /utils/runtime/update_chaos_result_resource.yml for 127.0.0.1

2020-06-30 14:55:24.654505 Step: [PreReq]: Generate the chaos result CR to reflect SOT (Start of Test) 
task path: /utils/runtime/update_chaos_result_resource.yml:3
details => {"changed": true, "checksum": "7ae8771d33cea8f5b48a294d34dece338ae49923", "dest": "/tmp/chaos-result.yml", "gid": 0, "group": "root", "md5sum": "ee6776809d2fdb0a35ac570e6c47a9c2", "mode": "0644", "owner": "root", "size": 335, "src": "/tmp/ansible-tmp-1593528924.42-228215993217195/source", "state": "file", "uid": 0}

2020-06-30 14:55:25.700432 Step: [PreReq]: Apply the chaos result CR for kubelet-service-kill experiment 
task path: /utils/runtime/update_chaos_result_resource.yml:13
details => {"changed": true, "cmd": "kubectl apply -f /tmp/chaos-result.yml -n default", "delta": "0:00:00.740294", "end": "2020-06-30 14:55:25.666973", "failed_when_result": false, "rc": 0, "start": "2020-06-30 14:55:24.926679", "stderr": "", "stderr_lines": [], "stdout": "chaosresult.litmuschaos.io/nginx-chaos-kubelet-service-kill configured", "stdout_lines": ["chaosresult.litmuschaos.io/nginx-chaos-kubelet-service-kill configured"]}

2020-06-30 14:55:25.871430 Step: [Info]: Display the application information passed via the test job 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:45
details => {
    "msg": [
        "The application info is as follows:", 
        "Namespace    : default", 
        "Label        : run=nginx", 
        "Ramp Time    : "
    ]
}
included: /utils/common/status_app_pod.yml for 127.0.0.1

2020-06-30 14:55:26.587508 Step: [Status]: Checking whether application pods are in running state 
task path: /utils/common/status_app_pod.yml:2
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pods -n default -l run=nginx -o custom-columns=:.status.phase --no-headers", "delta": "0:00:00.438530", "end": "2020-06-30 14:55:26.559277", "rc": 0, "start": "2020-06-30 14:55:26.120747", "stderr": "", "stderr_lines": [], "stdout": "Running\nRunning", "stdout_lines": ["Running", "Running"]}

2020-06-30 14:55:27.226374 Step: [Status]: Checking whether application containers are in running state 
task path: /utils/common/status_app_pod.yml:9
details => {"attempts": 1, "changed": true, "cmd": "kubectl get pod -n default -l run=nginx --no-headers -o jsonpath='{.items[*].status.containerStatuses[*].ready}' | tr ' ' '\\n' | uniq", "delta": "0:00:00.452996", "end": "2020-06-30 14:55:27.202578", "rc": 0, "start": "2020-06-30 14:55:26.749582", "stderr": "", "stderr_lines": [], "stdout": "true", "stdout_lines": ["true"]}
included: /utils/common/generate-kubernetes-chaos-events.yml for 127.0.0.1

2020-06-30 14:55:27.987793 Step: [Event]: Obtain the current date in ISO8601 format 
task path: /utils/common/generate-kubernetes-chaos-events.yml:1
details => {"changed": true, "cmd": "date --iso-8601=seconds", "delta": "0:00:00.341967", "end": "2020-06-30 14:55:27.959517", "rc": 0, "start": "2020-06-30 14:55:27.617550", "stderr": "", "stderr_lines": [], "stdout": "2020-06-30T14:55:27+00:00", "stdout_lines": ["2020-06-30T14:55:27+00:00"]}

2020-06-30 14:55:28.055438 Step: set_fact 
task path: /utils/common/generate-kubernetes-chaos-events.yml:5
details => {"ansible_facts": {"ts": "2020-06-30T14:55:27Z"}, "changed": false}

2020-06-30 14:55:28.890932 Step: [Event]: Creating a PreChaosCheck event in the chaosengine 
task path: /utils/common/generate-kubernetes-chaos-events.yml:8
details => {"changed": true, "method": "create", "result": {"apiVersion": "v1", "count": 1, "eventTime": null, "firstTimestamp": "2020-06-30T14:55:27Z", "involvedObject": {"apiVersion": "litmuschaos.io/v1alpha1", "kind": "ChaosEngine", "name": "nginx-chaos", "namespace": "default", "uid": "a1b6f7b9-d931-4c1e-96b7-c8f955b16144"}, "kind": "Event", "lastTimestamp": "2020-06-30T14:55:27Z", "message": "AUT is Running successfully", "metadata": {"creationTimestamp": "2020-06-30T14:55:28Z", "managedFields": [{"apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:count": {}, "f:firstTimestamp": {}, "f:involvedObject": {"f:apiVersion": {}, "f:kind": {}, "f:name": {}, "f:namespace": {}, "f:uid": {}}, "f:lastTimestamp": {}, "f:message": {}, "f:reason": {}, "f:source": {"f:component": {}}, "f:type": {}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2020-06-30T14:55:28Z"}], "name": "event-\"PreChaosCheck\"", "namespace": "default", "resourceVersion": "4453620", "selfLink": "/api/v1/namespaces/default/events/event-%22PreChaosCheck%22", "uid": "8c5881dc-f69c-447e-9672-0daa294d8427"}, "reason": "PreChaosCheck", "reportingComponent": "", "reportingInstance": "", "source": {"component": "kubelet-service-kill-rndfdt-sgvqw"}, "type": "Normal"}}
included: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml for 127.0.0.1

2020-06-30 14:55:29.719253 Step: [Prepare]: Select the application pod name 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:6
details => {"changed": true, "cmd": "kubectl get pod -l run=nginx -n default -o=custom-columns=:metadata.name --no-headers | shuf | head -1", "delta": "0:00:00.474022", "end": "2020-06-30 14:55:29.689615", "rc": 0, "start": "2020-06-30 14:55:29.215593", "stderr": "", "stderr_lines": [], "stdout": "nginx", "stdout_lines": ["nginx"]}

2020-06-30 14:55:29.788568 Step: [Prepare]: Recording the application pod name 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:15
details => {"ansible_facts": {"app_pod": "nginx"}, "changed": false}

2020-06-30 14:55:30.429653 Step: [Prepare]: Identify the application node name 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:21
details => {"changed": true, "cmd": "kubectl get pod nginx -n default --no-headers -o custom-columns=:spec.nodeName", "delta": "0:00:00.445249", "end": "2020-06-30 14:55:30.408565", "rc": 0, "start": "2020-06-30 14:55:29.963316", "stderr": "", "stderr_lines": [], "stdout": "shreyangi-w1", "stdout_lines": ["shreyangi-w1"]}

2020-06-30 14:55:30.495404 Step: [Prepare]: Record the application node name 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:29
details => {"ansible_facts": {"app_node": "shreyangi-w1"}, "changed": false}

2020-06-30 14:55:30.990231 Step: [Prepare:] Generate a run id if not passed from the engine/experiment 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:34
details => {"changed": true, "cmd": "echo $(mktemp) | cut -d '.' -f 2 | cut -c -6", "delta": "0:00:00.310400", "end": "2020-06-30 14:55:30.969843", "rc": 0, "start": "2020-06-30 14:55:30.659443", "stderr": "", "stderr_lines": [], "stdout": "kqvdqg", "stdout_lines": ["kqvdqg"]}

2020-06-30 14:55:31.061665 Step: set_fact 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:38
details => {"ansible_facts": {"run_id": "kqvdqg"}, "changed": false}
included: /utils/common/generate-kubernetes-chaos-events.yml for 127.0.0.1

2020-06-30 14:55:31.671582 Step: [Event]: Obtain the current date in ISO8601 format 
task path: /utils/common/generate-kubernetes-chaos-events.yml:1
details => {"changed": true, "cmd": "date --iso-8601=seconds", "delta": "0:00:00.329259", "end": "2020-06-30 14:55:31.651644", "rc": 0, "start": "2020-06-30 14:55:31.322385", "stderr": "", "stderr_lines": [], "stdout": "2020-06-30T14:55:31+00:00", "stdout_lines": ["2020-06-30T14:55:31+00:00"]}

2020-06-30 14:55:31.754564 Step: set_fact 
task path: /utils/common/generate-kubernetes-chaos-events.yml:5
details => {"ansible_facts": {"ts": "2020-06-30T14:55:31Z"}, "changed": false}

2020-06-30 14:55:32.425564 Step: [Event]: Creating a ChaosInject event in the chaosengine 
task path: /utils/common/generate-kubernetes-chaos-events.yml:8
details => {"changed": true, "method": "create", "result": {"apiVersion": "v1", "count": 1, "eventTime": null, "firstTimestamp": "2020-06-30T14:55:31Z", "involvedObject": {"apiVersion": "litmuschaos.io/v1alpha1", "kind": "ChaosEngine", "name": "nginx-chaos", "namespace": "default", "uid": "a1b6f7b9-d931-4c1e-96b7-c8f955b16144"}, "kind": "Event", "lastTimestamp": "2020-06-30T14:55:31Z", "message": "Injecting kubelet-service-kill chaos on shreyangi-w1 node", "metadata": {"creationTimestamp": "2020-06-30T14:55:32Z", "managedFields": [{"apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:count": {}, "f:firstTimestamp": {}, "f:involvedObject": {"f:apiVersion": {}, "f:kind": {}, "f:name": {}, "f:namespace": {}, "f:uid": {}}, "f:lastTimestamp": {}, "f:message": {}, "f:reason": {}, "f:source": {"f:component": {}}, "f:type": {}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2020-06-30T14:55:32Z"}], "name": "event-\"ChaosInject\"", "namespace": "default", "resourceVersion": "4453631", "selfLink": "/api/v1/namespaces/default/events/event-%22ChaosInject%22", "uid": "254cbe85-2347-4330-aaff-bef5a4966c46"}, "reason": "ChaosInject", "reportingComponent": "", "reportingInstance": "", "source": {"component": "kubelet-service-kill-rndfdt-sgvqw"}, "type": "Normal"}}

2020-06-30 14:55:32.753193 Step: [Prepare]: Patch the run_id to kubelet service kill helper pod template 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:51
details => {"changed": true, "checksum": "fa6857921f926bf8ee66b2f6d7aa43174d6ee002", "dest": "/tmp/kubelet-service-kill.yml", "gid": 0, "group": "root", "md5sum": "8f6c1f9e2190fac453dea74486552065", "mode": "0644", "owner": "root", "size": 847, "src": "/tmp/ansible-tmp-1593528932.5-27399575608010/source", "state": "file", "uid": 0}

2020-06-30 14:55:32.837116 Step: set_fact 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:59
details => {"ansible_facts": {"pod_running_status": ""}, "changed": false}
included: /utils/common/create_chaos_pod.yml for 127.0.0.1 => (item=1)
included: /utils/common/create_chaos_pod.yml for 127.0.0.1 => (item=2)
included: /utils/common/create_chaos_pod.yml for 127.0.0.1 => (item=3)

2020-06-30 14:55:33.711165 Step: [Inject]: Creating the chaos pod in default namespace 
task path: /utils/common/create_chaos_pod.yml:3
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "kubectl apply -f /tmp/kubelet-service-kill.yml -n default", "delta": "0:00:00.570614", "end": "2020-06-30 14:55:33.691480", "msg": "non-zero return code", "rc": 1, "start": "2020-06-30 14:55:33.120866", "stderr": "error: error parsing /tmp/kubelet-service-kill.yml: error converting YAML to JSON: yaml: line 8: mapping values are not allowed in this context", "stderr_lines": ["error: error parsing /tmp/kubelet-service-kill.yml: error converting YAML to JSON: yaml: line 8: mapping values are not allowed in this context"], "stdout": "", "stdout_lines": []}

2020-06-30 14:55:33.862050 Step: fail 
task path: /chaoslib/litmus/kubelet_service_kill/kubelet_service_kill.yml:142
fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "kubelet_service_kill lib failed"}

2020-06-30 14:55:33.922490 Step: set_fact 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:139
details => {"ansible_facts": {"flag": "Fail"}, "changed": false}

2020-06-30 14:55:34.004306 Step: [Result]: Getting the final result of kubelet-service-kill experiment 
task path: /experiments/generic/kubelet_service_kill/kubelet_service_kill_ansible_logic.yml:142
details => {
    "msg": "kubelet-service-kill experiment has been Failed"
}
included: /utils/runtime/getting_failure_step.yml for 127.0.0.1

2020-06-30 14:55:34.214722 Step: [Failure-Detection]: Recording the offset, on the basis of verbosity 
task path: /utils/runtime/getting_failure_step.yml:8
details => {"ansible_facts": {"offset": 2}, "changed": false}

2020-06-30 14:55:34.848452 Step: [Failure-Detection]: Getting name of failure step from experiment pod 
task path: /utils/runtime/getting_failure_step.yml:14
details => {"changed": true, "cmd": "kubectl logs kubelet-service-kill-rndfdt-sgvqw -n default | grep \"FAILED!\" -B 2 | head -1 | awk -F \"Step:\" '{print $2}'", "delta": "0:00:00.451220", "end": "2020-06-30 14:55:34.828776", "rc": 0, "start": "2020-06-30 14:55:34.377556", "stderr": "", "stderr_lines": [], "stdout": " [Inject]: Creating the chaos pod in default namespace ", "stdout_lines": [" [Inject]: Creating the chaos pod in default namespace "]}

2020-06-30 14:55:34.954166 Step: [Failure-Detection]: Recording the name of failed step 
task path: /utils/runtime/getting_failure_step.yml:21
details => {"ansible_facts": {"failStep": " [Inject]: Creating the chaos pod in default namespace "}, "changed": false}

2020-06-30 14:55:35.022962 Step: [Failure-Detection]: Printing the name of failed step 
task path: /utils/runtime/getting_failure_step.yml:25
details => {
    "msg": "failStep:  [Inject]: Creating the chaos pod in default namespace "
}
included: /utils/common/generate-kubernetes-chaos-events.yml for 127.0.0.1

2020-06-30 14:55:35.577645 Step: [Event]: Obtain the current date in ISO8601 format 
task path: /utils/common/generate-kubernetes-chaos-events.yml:1
details => {"changed": true, "cmd": "date --iso-8601=seconds", "delta": "0:00:00.317537", "end": "2020-06-30 14:55:35.554256", "rc": 0, "start": "2020-06-30 14:55:35.236719", "stderr": "", "stderr_lines": [], "stdout": "2020-06-30T14:55:35+00:00", "stdout_lines": ["2020-06-30T14:55:35+00:00"]}

2020-06-30 14:55:35.652553 Step: set_fact 
task path: /utils/common/generate-kubernetes-chaos-events.yml:5
details => {"ansible_facts": {"ts": "2020-06-30T14:55:35Z"}, "changed": false}

2020-06-30 14:55:36.288559 Step: [Event]: Creating a Summary event in the chaosengine 
task path: /utils/common/generate-kubernetes-chaos-events.yml:8
details => {"changed": true, "method": "create", "result": {"apiVersion": "v1", "count": 1, "eventTime": null, "firstTimestamp": "2020-06-30T14:55:35Z", "involvedObject": {"apiVersion": "litmuschaos.io/v1alpha1", "kind": "ChaosEngine", "name": "nginx-chaos", "namespace": "default", "uid": "a1b6f7b9-d931-4c1e-96b7-c8f955b16144"}, "kind": "Event", "lastTimestamp": "2020-06-30T14:55:35Z", "message": "kubelet-service-kill Experiment Failed!", "metadata": {"creationTimestamp": "2020-06-30T14:55:36Z", "managedFields": [{"apiVersion": "v1", "fieldsType": "FieldsV1", "fieldsV1": {"f:count": {}, "f:firstTimestamp": {}, "f:involvedObject": {"f:apiVersion": {}, "f:kind": {}, "f:name": {}, "f:namespace": {}, "f:uid": {}}, "f:lastTimestamp": {}, "f:message": {}, "f:reason": {}, "f:source": {"f:component": {}}, "f:type": {}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2020-06-30T14:55:36Z"}], "name": "event-\"Summary\"", "namespace": "default", "resourceVersion": "4453643", "selfLink": "/api/v1/namespaces/default/events/event-%22Summary%22", "uid": "59930c2b-05db-42fc-ab6c-6ab149da0895"}, "reason": "Summary", "reportingComponent": "", "reportingInstance": "", "source": {"component": "kubelet-service-kill-rndfdt-sgvqw"}, "type": "Normal"}}
included: /utils/runtime/update_chaos_result_resource.yml for 127.0.0.1

2020-06-30 14:55:37.184139 Step: [Result]: Update the chaos result CR to reflect EOT (End of Test) 
task path: /utils/runtime/update_chaos_result_resource.yml:23
details => {"changed": true, "checksum": "edaf58d6f1a93f4d5d9d0611154790b353127087", "dest": "/tmp/chaos-result.yml", "gid": 0, "group": "root", "md5sum": "20d496f0e27fef61738490e80349366a", "mode": "0644", "owner": "root", "size": 406, "src": "/tmp/ansible-tmp-1593528936.6-215650603947121/source", "state": "file", "uid": 0}

2020-06-30 14:55:38.103873 Step: [The END]: Apply the chaos result CR for kubelet-service-kill experiment 
task path: /utils/runtime/update_chaos_result_resource.yml:33
details => {"changed": true, "cmd": "kubectl apply -f /tmp/chaos-result.yml -n default", "delta": "0:00:00.662641", "end": "2020-06-30 14:55:38.076726", "failed_when_result": false, "rc": 0, "start": "2020-06-30 14:55:37.414085", "stderr": "", "stderr_lines": [], "stdout": "chaosresult.litmuschaos.io/nginx-chaos-kubelet-service-kill configured", "stdout_lines": ["chaosresult.litmuschaos.io/nginx-chaos-kubelet-service-kill configured"]}
META: ran handlers
META: ran handlers

2020-06-30 14:55:38.110150 ************ RELAX, EXPERIMENT ENDS! ************ 
127.0.0.1                  : ok=45   changed=18   unreachable=0    failed=2

Expected Output
Deployment of the experiment without any fail
@Shreyangi Shreyangi added the bug label Jul 1, 2020
@ksatchit ksatchit added this to the 1.5.1 milestone Jul 1, 2020
@uditgaurav
Copy link
Member

uditgaurav commented Jul 1, 2020 •
edited
Loading

Thanks, @Shreyangi for bringing this in a notice. This issue is resolved with litmuschaos/chaos-charts#249 and #158 the changes will be brought with 1.5.1 till then you can use ci tag.

@ksatchit
Copy link
Member

ksatchit commented Jul 2, 2020

Closing this issue as the fix will be available (both in experiment code as well as docs in 1.5.1)

@ksatchit ksatchit closed this as completed Jul 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@uditgaurav uditgaurav

Labels
bug kind/bug
Projects
None yet
Milestone
1.5.1
Development

No branches or pull requests
3 participants
@ksatchit @uditgaurav @Shreyangi