Sub-handlers in a resume handlers are executed not in parallel

[kopf-archiver]

An issue by e0ne at 2019-10-24 12:02:24+00:00
Original URL: zalando-incubator/kopf#214
 

Expected Behavior

Sub-handlers in a 'resume' event should be executed in parallel.

Actual Behavior

I've got one function both for create/update and resume events handling. create/update sub-handlers work in parallel without any issues. The same code for a resume event handler is executed one-by-one.

Steps to Reproduce the Problem

1. I've got some handlers like:

import functools
import kopf

@kopf.on.update('zalando.org', 'v1', 'kopfexamples')
@kopf.on.create('zalando.org', 'v1', 'kopfexamples')
def create_fn(spec, **_):
    fns = {}

    for item in spec.get('items', []):
        fns[item] = functools.partial(handle_item, item=item)

   kopf.execute(fns)

def handle_item(item, *, spec, **_):
    pass

2. I use CRD from examples
3. Start operator, apply crd.yaml and obj.yaml from examples
4. Restart operator.

Specifications

• Platform: Ubuntu 18.04
• Kubernetes version: 1.14.6
• Python version: 3.7.5
• Python packages installed: kopf from master

---

Commented by nolar at 2019-10-30 23:12:19+00:00
 

Hm. That is interesting. e0ne Can you please describe ho do you check that the handlers are executed in parallel?

The handlers for one single object are never executed in parallel, they are always sequential by design. More on that, it is not possible that the creation & update events happen at the same time (for update to happen, the object must be created already).

Multiple different objects are handled in parallel though (each with its own sequence of handlers).

This is also valid for the sub-handlers: they are executed in sequence (not necessary in the order given, but at most one at a time), and the original top-level handler is waiting for them.

So, it is not clear what do you mean that create/update are executed in parallel.

---

Also, I see that you call kopf.execute() directly. This will not work at all — the sub-handlers will not be executed. kopf.execute() is a coroutine, not a function, so it must be awaited.

I assume, you have taken it from the docs at https://kopf.readthedocs.io/en/latest/handlers/, don't you? It seems, the example there is wrong. Another page is more correct on await kopf.execute(...): https://kopf.readthedocs.io/en/latest/idempotence/ I will fix it soon.

Notice, that for await to be usable, the whole function must be declared as async def, and it is better if it is really async and non-blocking then (regular async/await development practices).

---

If actual parallel execution (but not handling) is needed, it is better to use the Python's parallelisation with threads or asyncio tasks.

Just as a rough idea:

import asyncio
import kopf

@kopf.on.update('zalando.org', 'v1', 'kopfexamples')
@kopf.on.create('zalando.org', 'v1', 'kopfexamples')
async def create_fn(spec, **kwargs):
    tasks = {}
    for item in spec.get('items', []):
        tasks[item] = asyncio.create_task(handle_item(item=item, spec=spec, **kwargs))
    done, pending = asyncio.wait(list(tasks.values()))

async def handle_item(item, *, spec, **_):
    pass

Same for threads and regular def functions.

The specific implementations are outside of the Kopf's scope, so this parallelisation pattern is not a part of it.

---

Commented by nolar at 2019-11-13 17:34:00+00:00
 

With kopf>=0.23rc1 (see the release notes), the on-resume handlers are fixed in few aspects:

• There can be more than one on-resume handler (previously only the first one was executed).
• They can go after the on-create/on-update handlers (previous worked only if they were the first in row).
• Arbitrary or temporary errors in the on-resume handlers are now retried, as for all other handlers (previously, were ignored).
• The sub-handlers should now be possible too (I didn't check though — but the preventing issue is the same as for all of the above).
• And they are not repeated anymore, once the object was resumed once.

See #230 for details.

I assume, the issue is fixed. The "parallel execution" issue is explained above — it never existed, and, perhaps, is just a terminology misunderstanding.

If the issue with the on-resume handlers continues to reappear in >=0.23rc1, feel free to re-open this issue with a describing comment (or at least just leave a comment).