Questions on running..... #80

bmos299 · 2021-04-07T18:59:11Z

I have some general questions.

I want to run Kraken in an openshift cluster and point it to another cluster for the chaos testing.

How do I get the .kubeconfig when using OpenShift?
In the config.yaml. Why is there two different pod_scenarios sections? Could this be all under one?

        -   pod_scenarios:                                 # List of chaos pod scenarios to load
            - -    scenarios/etcd.yml
            - -    scenarios/regex_openshift_pod_kill.yml
              -    scenarios/post_action_regex.py
        -   node_scenarios:                                # List of chaos node scenarios to load
            -   scenarios/node_scenarios_example.yml
        -   pod_scenarios:
            - -    scenarios/openshift-apiserver.yml
            - -    scenarios/openshift-kube-apiserver.yml

Thanks.

The text was updated successfully, but these errors were encountered:

paigerube14 · 2021-04-07T20:13:10Z

Great questions!!

So the .kubeconfig file is normally at ${HOME}/.kube/config or if the environment variable is set, it normally would be $KUBECONFIG
In the config.yaml file, yes the pod_scenarios sections could be under one. We are just trying to show that it is possible to execute the different types of scenarios in multiple different orders and also multiple times.

Hope this helps, please let me know if any of this doesn't make sense

bmos299 · 2021-04-08T14:52:53Z

Thanks @paigerube14. I ran the deploy (kraken.yml) and the deploy worked and the pod came up but quickly crashed in python. I looked at the logs and see this. Any thoughts? Also, if it did work....is there where I would go to see the run and results?

Barrys-MBP:containers bmosus.ibm.com$ oc get po
NAME                                 READY   STATUS    RESTARTS   AGE
kraken-deployment-5d9f5c77d9-gg56l   1/1     Running   5          4m38s
Barrys-MBP:containers bmosus.ibm.com$ oc logs kraken-deployment-5d9f5c77d9-gg56l
2021-04-08 14:48:35,983 [INFO] Starting kraken
2021-04-08 14:48:35,989 [INFO] Initializing client to talk to the Kubernetes cluster
 _              _              
| | ___ __ __ _| | _____ _ __  
| |/ / '__/ _` | |/ / _ \ '_ \ 
|   <| | | (_| |   <  __/ | | |
|_|\_\_|  \__,_|_|\_\___|_| |_|
                               

Traceback (most recent call last):
  File "run_kraken.py", line 403, in <module>
    main(options.cfg)
  File "run_kraken.py", line 309, in main
    kubecli.find_kraken_node()
  File "/root/kraken/kraken/kubernetes/client.py", line 180, in find_kraken_node
    pod_json = json.loads(pod_json_str)
  File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

paigerube14 · 2021-04-08T18:31:15Z

I usually don't run it using the deploy; I run using a locally cloned version. I know we have been having issues using the deploy (see here: #79) we are working on trying to get a fix for that.

Normally (if things worked) you would see the run and results right in the logs there that you are looking at or in the terminal, if running locally. It'll print off which scenario it is running and the results ( see below for example)


% python run_kraken.py 
 _              _              
| | ___ __ __ _| | _____ _ __  
| |/ / '__/ _` | |/ / _ \ '_ \ 
|   <| | | (_| |   <  __/ | | |
|_|\_\_|  \__,_|_|\_\___|_| |_|
                               

2021-04-08 10:59:07,010 [INFO] Starting kraken
2021-04-08 10:59:07,016 [INFO] Initializing client to talk to the Kubernetes cluster
2021-04-08 10:59:10,910 [INFO] Fetching cluster info
2021-04-08 10:59:11,651 [INFO] 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-06-162113   True        False         22h     Cluster version is 4.8.0-0.nightly-2021-04-06-162113
Kubernetes control plane is running at https://api.qe-pr-aws9.qe.devcluster.openshift.com:6443

2021-04-08 10:59:11,652 [INFO] Daemon mode not enabled, will run through 1 iterations

2021-04-08 10:59:11,652 [INFO] Executing scenarios for iteration 0
2021-04-08 10:59:16 INFO __main__ verbosity: None; log level: INFO; handler level: INFO
2021-04-08 10:59:16 INFO __main__ Creating kubernetes client with config /Users/prubenda/.kube/config from --kubeconfig flag
2021-04-08 10:59:16 INFO k8s_client Initializing with config: /Users/prubenda/.kube/config
2021-04-08 10:59:16 INFO __main__ No cloud driver - some functionality disabled
2021-04-08 10:59:16 INFO __main__ Using stdout metrics collector
2021-04-08 10:59:16 INFO __main__ NOT starting the UI server
2021-04-08 10:59:16 INFO __main__ STARTING AUTONOMOUS MODE
2021-04-08 10:59:18 INFO scenario.delete etcd pod Starting scenario 'delete etcd pods' (2 steps)
2021-04-08 10:59:18 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matched 3 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Initial set length: 3
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 1
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Pod killed: [pod #1 name=etcd-ip-10-0-168-189.us-east-2.compute.internal namespace=openshift-etcd containers=3 ip=10.0.168.189 host_ip=10.0.168.189 state=Running labels:app=etcd,etcd=true,k8s-app=etcd,revision=3 annotations:kubernetes.io/config.hash=5a0603be-bcf3-485e-a99d-8a25d514a145,kubernetes.io/config.mirror=5a0603be-bcf3-485e-a99d-8a25d514a145,kubernetes.io/config.seen=2021-04-07T16:02:58.149983648Z,kubernetes.io/config.source=file]
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 10:59:19 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 10:59:19 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 10:59:49 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 10:59:49 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 11:00:19 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 11:00:19 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Matched 3 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Initial set length: 3
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Filtered set length: 3
2021-04-08 11:00:49 INFO scenario.delete etcd pod Scenario finished
2021-04-08 11:00:49 INFO policy_runner All done here!

2021-04-08 11:00:50,070 [INFO] Scenario: scenarios/etcd.yml has been successfully injected!
2021-04-08 11:00:50,071 [INFO] Waiting for the specified duration: 60
2021-04-08 11:01:50,073 [INFO]

bmos299 · 2021-04-08T19:24:20Z

I think you need a requirement that allow one to add an openshift login. Something like,
oc login --token=sha256~IZx_NQ8F5GFBVtaj6blabblahblahGmf5IqI9FuiMWOMzs --server=https://api.pipeline.cp.barry.ibm.com:6443

bmos299 · 2021-04-08T19:27:00Z

@paigerube14 it looks like you co-authored the blog i am using, https://www.openshift.com/blog/introduction-to-kraken-a-chaos-tool-for-openshift/kubernetes. I am just trying to find a simple way to use a .kubeconfig with openshift as I usually just login with the token.

mffiedler · 2021-04-08T20:04:07Z

The kube config is generated a couple of ways. After install, the system:admin kubeconfig for OCP installs is in <install_dir>/auth/kubeconfig. If you are using oc login -u or oc login -t it will be in <home_dir>/.kube/config unless you are using the "oc config" command to do advanced context switching.

It sounds like your issue is with getting a kube config for a separate cluster into the cluster where Kraken is running. Have you tried creating a ConfigMap with a valid kube config for the remote cluster you are trying to access with Kraken? See https://github.com/cloud-bulldozer/kraken/tree/master/containers. I would also suggest trying to run Kraken non-containerized (from a git clone) first and then migrating a working config to ConfigMaps and containers. Let me know if I am mis-understanding the issue.

bmos299 · 2021-04-08T20:19:30Z

@paigerube14 and @mffiedler thank you. Trying now.

bmos299 · 2021-04-09T12:44:30Z

I have it running now. It looks like it is exiting due to the cloud type in node_scenarios: cloud_type:.

It looks like the cloud can be set to aws, gcp, openstack, and azure. I have two environments I run on. The first is an ocp stood up internally and the other is the ROKS on the IBM cloud. Are these environments supported?

chaitanyaenr · 2021-04-11T13:15:22Z

@bmos299 The current supported environments include AWS, GCP, OpenStack and Azure like you mentioned: https://github.com/cloud-bulldozer/kraken/blob/master/docs/node_scenarios.md. @jaredoconnell has a PR open which adds baremetal support.

There are just a couple of node scenarios like reboot/stop etc. that depends on the cloud on top of which OCP is running since they use the cloud API for the operation, the pod based scenarios, time based scenarios and a couple of node based scenarios including the node crash ( fork bomb ) and kubelet reboot can run independent of the cloud platform. Also, Litmus scenarios can also be triggered using Kraken: https://github.com/cloud-bulldozer/kraken/blob/master/docs/litmus_scenarios.md.

Can you create an issue for IBM cloud when you get time please? This way it will be under our radar.

bmos299 · 2021-04-12T12:55:12Z

@chaitanyaenr I see that @seanogor opened #84 . Is this the requirement you need?

chaitanyaenr · 2021-04-14T11:44:59Z

@bmos299 Yes, the issues opened by @seanogor should help with tracking it. We would definitely appreciate any help with it. Thanks.

seanogor · 2021-04-14T11:56:17Z

I usually don't run it using the deploy; I run using a locally cloned version. I know we have been having issues using the deploy (see here: #79) we are working on trying to get a fix for that.

Normally (if things worked) you would see the run and results right in the logs there that you are looking at or in the terminal, if running locally. It'll print off which scenario it is running and the results ( see below for example)


% python run_kraken.py 
 _              _              
| | ___ __ __ _| | _____ _ __  
| |/ / '__/ _` | |/ / _ \ '_ \ 
|   <| | | (_| |   <  __/ | | |
|_|\_\_|  \__,_|_|\_\___|_| |_|
                               

2021-04-08 10:59:07,010 [INFO] Starting kraken
2021-04-08 10:59:07,016 [INFO] Initializing client to talk to the Kubernetes cluster
2021-04-08 10:59:10,910 [INFO] Fetching cluster info
2021-04-08 10:59:11,651 [INFO] 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-06-162113   True        False         22h     Cluster version is 4.8.0-0.nightly-2021-04-06-162113
Kubernetes control plane is running at https://api.qe-pr-aws9.qe.devcluster.openshift.com:6443

2021-04-08 10:59:11,652 [INFO] Daemon mode not enabled, will run through 1 iterations

2021-04-08 10:59:11,652 [INFO] Executing scenarios for iteration 0
2021-04-08 10:59:16 INFO __main__ verbosity: None; log level: INFO; handler level: INFO
2021-04-08 10:59:16 INFO __main__ Creating kubernetes client with config /Users/prubenda/.kube/config from --kubeconfig flag
2021-04-08 10:59:16 INFO k8s_client Initializing with config: /Users/prubenda/.kube/config
2021-04-08 10:59:16 INFO __main__ No cloud driver - some functionality disabled
2021-04-08 10:59:16 INFO __main__ Using stdout metrics collector
2021-04-08 10:59:16 INFO __main__ NOT starting the UI server
2021-04-08 10:59:16 INFO __main__ STARTING AUTONOMOUS MODE
2021-04-08 10:59:18 INFO scenario.delete etcd pod Starting scenario 'delete etcd pods' (2 steps)
2021-04-08 10:59:18 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matched 3 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Initial set length: 3
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 1
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Pod killed: [pod #1 name=etcd-ip-10-0-168-189.us-east-2.compute.internal namespace=openshift-etcd containers=3 ip=10.0.168.189 host_ip=10.0.168.189 state=Running labels:app=etcd,etcd=true,k8s-app=etcd,revision=3 annotations:kubernetes.io/config.hash=5a0603be-bcf3-485e-a99d-8a25d514a145,kubernetes.io/config.mirror=5a0603be-bcf3-485e-a99d-8a25d514a145,kubernetes.io/config.seen=2021-04-07T16:02:58.149983648Z,kubernetes.io/config.source=file]
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 10:59:19 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 10:59:19 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 10:59:49 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 10:59:49 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 11:00:19 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 11:00:19 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Matched 3 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Initial set length: 3
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Filtered set length: 3
2021-04-08 11:00:49 INFO scenario.delete etcd pod Scenario finished
2021-04-08 11:00:49 INFO policy_runner All done here!

2021-04-08 11:00:50,070 [INFO] Scenario: scenarios/etcd.yml has been successfully injected!
2021-04-08 11:00:50,071 [INFO] Waiting for the specified duration: 60
2021-04-08 11:01:50,073 [INFO]

How do you enable this level of debugging, can you pass in a parameter in the cerberus/kraken configuration maps?

seanogor · 2021-04-14T11:57:40Z

@bmos299 Yes, the issues opened by @seanogor should help with tracking it. We would definitely appreciate any help with it. Thanks.

Will open a case to track, if you have an IBM contact we can add to the case...

chaitanyaenr · 2021-04-14T12:27:52Z

I usually don't run it using the deploy; I run using a locally cloned version. I know we have been having issues using the deploy (see here: #79) we are working on trying to get a fix for that.
Normally (if things worked) you would see the run and results right in the logs there that you are looking at or in the terminal, if running locally. It'll print off which scenario it is running and the results ( see below for example)


% python run_kraken.py 
 _              _              
| | ___ __ __ _| | _____ _ __  
| |/ / '__/ _` | |/ / _ \ '_ \ 
|   <| | | (_| |   <  __/ | | |
|_|\_\_|  \__,_|_|\_\___|_| |_|
                               

2021-04-08 10:59:07,010 [INFO] Starting kraken
2021-04-08 10:59:07,016 [INFO] Initializing client to talk to the Kubernetes cluster
2021-04-08 10:59:10,910 [INFO] Fetching cluster info
2021-04-08 10:59:11,651 [INFO] 
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-06-162113   True        False         22h     Cluster version is 4.8.0-0.nightly-2021-04-06-162113
Kubernetes control plane is running at https://api.qe-pr-aws9.qe.devcluster.openshift.com:6443

2021-04-08 10:59:11,652 [INFO] Daemon mode not enabled, will run through 1 iterations

2021-04-08 10:59:11,652 [INFO] Executing scenarios for iteration 0
2021-04-08 10:59:16 INFO __main__ verbosity: None; log level: INFO; handler level: INFO
2021-04-08 10:59:16 INFO __main__ Creating kubernetes client with config /Users/prubenda/.kube/config from --kubeconfig flag
2021-04-08 10:59:16 INFO k8s_client Initializing with config: /Users/prubenda/.kube/config
2021-04-08 10:59:16 INFO __main__ No cloud driver - some functionality disabled
2021-04-08 10:59:16 INFO __main__ Using stdout metrics collector
2021-04-08 10:59:16 INFO __main__ NOT starting the UI server
2021-04-08 10:59:16 INFO __main__ STARTING AUTONOMOUS MODE
2021-04-08 10:59:18 INFO scenario.delete etcd pod Starting scenario 'delete etcd pods' (2 steps)
2021-04-08 10:59:18 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matched 3 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Initial set length: 3
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 1
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Pod killed: [pod #1 name=etcd-ip-10-0-168-189.us-east-2.compute.internal namespace=openshift-etcd containers=3 ip=10.0.168.189 host_ip=10.0.168.189 state=Running labels:app=etcd,etcd=true,k8s-app=etcd,revision=3 annotations:kubernetes.io/config.hash=5a0603be-bcf3-485e-a99d-8a25d514a145,kubernetes.io/config.mirror=5a0603be-bcf3-485e-a99d-8a25d514a145,kubernetes.io/config.seen=2021-04-07T16:02:58.149983648Z,kubernetes.io/config.source=file]
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 10:59:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 10:59:19 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 10:59:19 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 10:59:49 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 10:59:49 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 10:59:49 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Matched 2 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Initial set length: 2
2021-04-08 11:00:19 INFO action_nodes_pods.delete etcd pod Filtered set length: 2
2021-04-08 11:00:19 ERROR action_nodes_pods.delete etcd pod Expected 3 pods, got 2
2021-04-08 11:00:19 WARNING scenario.delete etcd pod Failure in action. Sleeping 30 and retrying
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Matching 'labels' {'labels': {'namespace': 'openshift-etcd', 'selector': 'k8s-app=etcd'}}
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Matched 3 pods for selector k8s-app=etcd in namespace openshift-etcd
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Initial set length: 3
2021-04-08 11:00:49 INFO action_nodes_pods.delete etcd pod Filtered set length: 3
2021-04-08 11:00:49 INFO scenario.delete etcd pod Scenario finished
2021-04-08 11:00:49 INFO policy_runner All done here!

2021-04-08 11:00:50,070 [INFO] Scenario: scenarios/etcd.yml has been successfully injected!
2021-04-08 11:00:50,071 [INFO] Waiting for the specified duration: 60
2021-04-08 11:01:50,073 [INFO]

How do you enable this level of debugging, can you pass in a parameter in the cerberus/kraken configuration maps?

The logs of the pod should print the detailed information of the run @bmos299. if not we definitely need to fix it :-)

chaitanyaenr · 2021-04-14T12:29:41Z

Will open a case to track, if you have an IBM contact we can add to the case...

That would to great @seanogor, we don't have an IBM contact.

seanogor · 2021-04-14T14:10:26Z

Could you comment on the debugging I mentioned?

chaitanyaenr · 2021-04-15T19:22:38Z

@seanogor The logs have the debugging information: 1) standalone kraken run will print the logs to the stdout 2) podman/docker logs -f can be used to stream the logs in case of containerized version 3) oc/kubectl logs -f can be used to stream the logs.

bmos299 · 2021-04-27T13:55:00Z

Thanks for the help. Closing.

seanogor mentioned this issue Apr 12, 2021

ROKS node support #84

Open

bmos299 closed this as completed Apr 27, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Questions on running..... #80

Questions on running..... #80

bmos299 commented Apr 7, 2021 •

edited

paigerube14 commented Apr 7, 2021

bmos299 commented Apr 8, 2021

paigerube14 commented Apr 8, 2021

bmos299 commented Apr 8, 2021

bmos299 commented Apr 8, 2021

mffiedler commented Apr 8, 2021

bmos299 commented Apr 8, 2021

bmos299 commented Apr 9, 2021 •

edited

chaitanyaenr commented Apr 11, 2021

bmos299 commented Apr 12, 2021

chaitanyaenr commented Apr 14, 2021

seanogor commented Apr 14, 2021

seanogor commented Apr 14, 2021

chaitanyaenr commented Apr 14, 2021

chaitanyaenr commented Apr 14, 2021

seanogor commented Apr 14, 2021

chaitanyaenr commented Apr 15, 2021

bmos299 commented Apr 27, 2021

Questions on running..... #80

Questions on running..... #80

Comments

bmos299 commented Apr 7, 2021 • edited

paigerube14 commented Apr 7, 2021

bmos299 commented Apr 8, 2021

paigerube14 commented Apr 8, 2021

bmos299 commented Apr 8, 2021

bmos299 commented Apr 8, 2021

mffiedler commented Apr 8, 2021

bmos299 commented Apr 8, 2021

bmos299 commented Apr 9, 2021 • edited

chaitanyaenr commented Apr 11, 2021

bmos299 commented Apr 12, 2021

chaitanyaenr commented Apr 14, 2021

seanogor commented Apr 14, 2021

seanogor commented Apr 14, 2021

chaitanyaenr commented Apr 14, 2021

chaitanyaenr commented Apr 14, 2021

seanogor commented Apr 14, 2021

chaitanyaenr commented Apr 15, 2021

bmos299 commented Apr 27, 2021

bmos299 commented Apr 7, 2021 •

edited

bmos299 commented Apr 9, 2021 •

edited