Operator freezes while exiting after an error in the watching/queueing cycle

[kopf-archiver]

An issue by nolar at 2019-04-08 12:13:15+00:00
Original URL: zalando-incubator/kopf#25
 

Expected Behavior

When an error happens in the watching/queueing coroutines, the process exits, and the pod is restarted by Kubernetes (or it just exists if executed locally).

Actual Behavior

In some cases, the process freezes after the exception, and no new events are handled, nothing is logged.

Steps to Reproduce the Problem

Uncertain, but:

1. Simulate an error in the watching cycle, e.g. such as Kopf-based operator fails with KeyError ['uid'] #10

---

Commented by nolar at 2019-04-16 14:59:09+00:00
 

The issue is presumably fixed in #27.

"Presumably" means that it was one certain way of freezing the operator forever with no reaction, and the simulated symptoms match with the observed symptoms. This way is now fixed.

But it is unclear how this way could be triggered and activated: the kubernetes.watch.Watch().stream() call never ends normally, since it has while True inside.

Which, in turn, means that there could be other reasons and ways of freezing. We need to catch them first, and investigate if it happens again.