Added support to track pod crash/restart during sleep #52
Conversation
|
Please have a look @chaitanyaenr @mffiedler |
|
Can one of the admins verify this patch? |
start_cerberus.py
Outdated
| @@ -165,6 +167,10 @@ def main(cfg): | |||
| watch_namespaces_status, failed_nodes, | |||
| failed_pods_components) | |||
|
|
|||
| if iteration != 1 and crashed_restarted_pods: | |||
| logging.info("Pods that were crashed/restarted during the sleep: %s" | |||
There was a problem hiding this comment.
Suggestion: "Pods that crashed/restarted during iteration %s : %s" % (iteration, crashed_restarted_pods)
cerberus/kubernetes/client.py
Outdated
| pods_tracker[pod]["creation_timestamp"] = pod_creation_timestamp | ||
| pods_tracker[pod]["restart_count"] = pod_restart_count | ||
| else: | ||
| crashed_restarted_pods.append(pod) |
There was a problem hiding this comment.
The list of crashed/restarted pods is currently the pod name. A possible improvement is to append <pod_namspace>:<pod_name> to the list to make it clear where the pod is. This is minor though, I am fine if we move forward like it is now.
start_cerberus.py
Outdated
| @@ -165,6 +167,10 @@ def main(cfg): | |||
| watch_namespaces_status, failed_nodes, | |||
| failed_pods_components) | |||
|
|
|||
| if iteration != 1 and crashed_restarted_pods: | |||
| logging.info("Pods that were crashed/restarted during the sleep: %s" | |||
There was a problem hiding this comment.
Suggestion: "Pods that were crashed/restarted during iteration %s : %s" % (iteration, crashed_restarted_pods)"
|
Not sure why the duplicate comment. In any case my comments are nitpicks. I tested this and it looks good to me. |
cerberus/kubernetes/client.py
Outdated
|
|
||
| # Load kubeconfig and initialize kubernetes python client | ||
| def initialize_clients(kubeconfig_path): | ||
| global cli | ||
| global cli, pods_tracker |
There was a problem hiding this comment.
Maybe initialize pods_tracker outside this function as it's meant for initializing client?
cerberus/kubernetes/client.py
Outdated
| crashed_restarted_pods = [] | ||
| for pod in pods: | ||
| try: | ||
| pod_info = cli.read_namespaced_pod_status(pod, namespace, |
There was a problem hiding this comment.
Make this a separate function and call it to provide the given pod status? This way we can reuse it in other places as well. Thoughts?
cerberus/kubernetes/client.py
Outdated
| @@ -80,6 +82,41 @@ def check_sdn_namespace(): | |||
| please specify the correct networking namespace in config file") | |||
|
|
|||
|
|
|||
| def namespace_sleep_track(namespace): | |||
There was a problem hiding this comment.
Let's add a comment to describe the functionality of this function.
yashashreesuresh
force-pushed
the
sleep_tracker
branch
2 times, most recently
from
e664d2b
to
f22f53f
Compare
|
I have made all the above changes. PTAL @mffiedler @chaitanyaenr |
This commit enables the pod crash/restart to be tracked during the wait time between each iteration. This prevents the Cerberus from missing pod crash/restarts when the pod enters the running phase before the next iteration.
yashashreesuresh
force-pushed
the
sleep_tracker
branch
from
f22f53f
to
456aff2
Compare
|
Nice job @yashashreesuresh. |
This commit enables the pod crash/restart to be tracked during the wait time between each iteration. This prevents the Cerberus from missing pod crash/restarts when the pod enters the running phase before the next iteration.
Fixes: #51