[PR] Interrupt the sleep for the delayed handlers on new changes

[kopf-archiver]

A pull request by nolar at 2019-08-02 00:28:17+00:00
Original URL: zalando-incubator/kopf#162
Merged by nolar at 2019-08-08 12:00:10+00:00

Wake up and react immediately when new watch-events arrive while operator is sleeping until the next delayed/postponed handler.

Issue : #161, indirectly #142

Description

Previously, Kopf used asyncio.sleep() for shortest sleep duration until the next postponed handler. If new events arrived for that object — e.g. it was changed or deleted — the sleep continued unconditionally.

With this PR, the sleep until the next postponed handler will be interrupted as soon as a new event arrives (i.e. the object's event queue is replenished). Kopf will reconsider the next steps based on the new object's state, and either process the new state (e.g. finish handling the deleted object), or continue sleeping for the remaining time.

Both intended and remaining time are logged. For example:

[2019-08-02 02:21:21,414] kopf.objects         [DEBUG   ] [default/kopf-example-1] Creation event: ...
[2019-08-02 02:21:21,415] kopf.objects         [DEBUG   ] [default/kopf-example-1] Invoking handler 'create_fn'.
[2019-08-02 02:21:23,421] kopf.objects         [ERROR   ] [default/kopf-example-1] Handler 'create_fn' failed with an exception. Will retry.
Traceback (most recent call last):
  ...
  File "example.py", line 14, in create_fn
    raise Exception("First failure.")
Exception: First failure.
[2019-08-02 02:21:23,427] kopf.objects         [DEBUG   ] [default/kopf-example-1] Patching with: {'status': {'kopf': {'progress': {'create_fn': {'started': '2019-08-02T00:21:21.415865', 'retries': 1, 'delayed': '2019-08-02T00:22:23.427267'}}}}}
[2019-08-02 02:21:23,599] kopf.objects         [DEBUG   ] [default/kopf-example-1] Creation event: ...
[2019-08-02 02:21:23,601] kopf.objects         [DEBUG   ] [default/kopf-example-1] Sleeping for 59.825606 seconds for the delayed handlers.
[2019-08-02 02:21:29,755] kopf.objects         [DEBUG   ] [default/kopf-example-1] Sleeping was interrupted by new changes, 53.672287957 seconds left.
[2019-08-02 02:21:29,862] kopf.objects         [DEBUG   ] [default/kopf-example-1] Creation event: ...
[2019-08-02 02:21:29,864] kopf.objects         [DEBUG   ] [default/kopf-example-1] Sleeping for 53.562986 seconds for the delayed handlers.
[2019-08-02 02:21:45,136] kopf.objects         [DEBUG   ] [default/kopf-example-1] Sleeping was interrupted by new changes, 38.291081867 seconds left.
[2019-08-02 02:21:45,238] kopf.objects         [DEBUG   ] [default/kopf-example-1] Deleted, really deleted, and we are notified.

This speeds up the responsiveness of the operators in case of object changes.

Types of Changes

• Refactor/improvements

Review

• Tests
• Documentation