Kraken regex openshift kill pod fails for statefulset pods

[1kaushik1]

Hello, We are trying to execute kraken to simulate random pod kills on a specific namespace and we are using the regex pod kill scenario defined here
https://github.com/redhat-chaos/krkn/blob/v1.4.4/scenarios/openshift/regex_openshift_pod_kill.yml

What we can see is, it works as expected when the application pods are deployed as deployments but fails with a timeout when hey are deployed as statefulsets.

python3.9 run_kraken.py --config config/config_openshift.yaml
 _              _
| | ___ __ __ _| | _____ _ __
| |/ / '__/ _` | |/ / _ \ '_ \
|   <| | | (_| |   <  __/ | | |
|_|\_\_|  \__,_|_|\_\___|_| |_|


2023-08-25 08:20:17,348 [INFO] Starting kraken
2023-08-25 08:20:17,353 [INFO] Initializing client to talk to the Kubernetes cluster
2023-08-25 08:20:39,119 [INFO] Publishing kraken status at http://0.0.0.0:8082
2023-08-25 08:20:39,123 [INFO] Starting http server at http://0.0.0.0:8082

2023-08-25 08:20:39,124 [INFO] Fetching cluster info
2023-08-25 08:20:39,151 [INFO] Cluster version is 4.12.26
2023-08-25 08:20:39,151 [INFO] Server URL: https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxx:6443
2023-08-25 08:20:39,151 [INFO] Generated a uuid for the run: xxxxxxxxxxxxxxxxxx01df30e23350
2023-08-25 08:20:39,152 [INFO] Daemon mode not enabled, will run through 1 iterations

2023-08-25 08:20:39,152 [INFO] Executing scenarios for iteration 0
2023-08-25 08:20:39,152 [INFO] connection set up
127.0.0.1 - - [25/Aug/2023 08:20:39] "GET / HTTP/1.1" 200 -
2023-08-25 08:20:39,156 [INFO] response RUN
2023-08-25 08:20:59,943 [INFO] {
        "output_id": "success",
        "output_data": {
                "pods": {
                        "1692951658935322803": {
                                "namespace": "test-project1",
                                "name": "httpd-sample-57988489d7-fmgnl"
                        }
                }
        }
}

2023-08-25 08:21:21,070 [INFO] {
        "output_id": "success",
        "output_data": {
                "pods": [
                        {
                                "namespace": "test-project1",
                                "name": "httpd-sample-57988489d7-lxxtd"
                        }                                                                                                                          [102/1972]
                ]
        }
}

2023-08-25 08:23:41,587 [INFO] {
        "output_id": "error",
        "output_data": {
                "error": "Timeout while waiting for pods to be removed."
        }
}

2023-08-25 08:23:41,587 [ERROR] Error while running scenarios/openshift/regex_openshift_pod_kill.yml: Step 2 in scenarios/openshift/regex_openshift_pod_kill.
yml (kill-pods) failed
2023-08-25 08:23:41,588 [INFO]
2023-08-25 08:23:41,588 [ERROR] Post scenarios are still failing at the end of all iterations

CAn you let us kow ho we can deploy kraken pod kill scenarios for statefulsets ?

[jtydlack]

Hi, I apologise for taking so long to reply.

I had quite a hard time reproducing the error and I came to the conclusion, that it was fixed in the arcaflow-plugin-kill-pod in this commit, so if you are still having this issue make sure you have the latest version of it.

The problem was that the pod would come back up so quickly that it did not think that the pod had been killed.

I understand if you have already fixed it.

[chaitanyaenr]

Closing the issue as it should be fixed in the latest version like @juliemathew mentioned. Please feel free to re-open/reach out in case of any issues/questions/feedback. Thanks.