[PR] Handle only the latest event with the most up to date state

[kopf-archiver]

A pull request by nolar at 2019-04-26 08:41:51+00:00
Original URL: zalando-incubator/kopf#43
Merged by nolar at 2019-04-26 15:15:01+00:00

Issue : #42

With this change, if the queue contains a batch of events, react only to the latest one — to follow the "eventual consistency" principle of Kubernetes.

The batch is time-framed to 0.1s, so that it is fast enough for all normal cases when only one event arrives, but slow enough for the initial object listing when multiple events arrive.

Still, some minimal time-frame is needed, as the streaming and parsing of the events inside of the kubernetes library is not immediate.

---

Commented by samurang87 at 2019-04-26 12:58:44+00:00
 

So basically if the fetching manages to get any events, you restart it to make sure there is only one event fetched? 🤔

---

Commented by nolar at 2019-04-26 14:23:39+00:00
 

samurang87 Yes. And getting all the events till nothing is left. All events except the last one are therefore ignored — as intended.

---

Commented by parking52 at 2019-04-26 14:31:19+00:00
 

Why will the event we are interested in be fetched rapidly ? And why will the number of event decrease ?

---

Commented by nolar at 2019-04-26 14:38:54+00:00
 

parking52 samurang87 I didn't get your last comments. What number of events decreased? What exactly is fetched rapidly?

---

Commented by nolar at 2019-04-26 14:39:49+00:00
 

parking52 samurang87 See the examples in #42 — there are few objects with few events (e.g. "mycrd-expr1"), of which only the latest is of our interest.

---

Commented by nolar at 2019-04-26 15:14:04+00:00
 

parking52 samurang87 This is efficiently an equivalent of this logic:

while queue.qsize() > 0:
    event = await queue.get()

Except that:

• threading.Queue().qsize() on Mac OS (it is not supported), so I used to avoid qsize() and just to not think about it. It is promised to work normally in asyncio.Queue though.
• If the queue is empty (qsize == 0), we should wait for 5.0 seconds until something appears. So, while queue.qsize() > 0: is not a suitable criterion here, unless first_time is introduced. But it makes it same as not nice as double tries. In that case, it would look like this:

event = None
first_time = True
try:
    while first_time or queue.qsize() > 0:
        first_time = False
        event = await asyncio.wait_for(queue.get(), timeout=5.0)
except asyncio.TimeoutError:
    if event is None:
        break

# when (event is not None) or (no timeout on the last event), continue.