asyncio task dependency manager

[vkottler]

First we need to move the code from a branch on vmklib here.

We need to make sure we await on all of the dependencies at once and not just serially.

We could test this with a "sleeper" task and make sure one task with 3 one-second sleeps completes in less than 3 seconds.

We need to implement a task that calls subprocess.run and make sure that it correctly yields

[vkottler]

What would be a way to implement "clearing" (and re-triggering?) tasks that came before you?

I suppose the Task interface needs a reset method and the class run funtion needs an optional parent argument or something?

we can pass self here: https://github.com/vkottler/vmklib/blob/move-to-vcorelib/vmklib/task/__init__.py#L78

debug log what we were invoked by

add Optional[Task] to the execute method prototype and pass it to self.execute from dispatch

[vkottler]

We added a task manager class and individual tasks' dispatch method now provides a reference to the caller, so you could clear its completed flag and re-await if if desired.

Next up is to implement a subprocess.run wrapper, among other things...

could be useful for the task manager to build out a dependency graph (or do like a dry run?)

might be worth giving dispatch a dry-run argument so that they can just return without doing anything, but also not set itself as completed (could be useful when looking at debug log)

[vkottler]

dependencies should be a set (de-duplication), finalized should always get cleared when a task or any dependencies are added

[vkottler]

eventual support for pattern matching tasks?

[vkottler]

Next up:

could be useful for the task manager to build out a dependency graph

Agree, now that we have support for init_only it would be cool to have a bit more debugging information. When we execute I would guess that we can build distinct "dependency trees" for each top-level task requested, not sure how to do this

Next up is to implement a subprocess.run wrapper, among other things...

Yes, since this should be one of the more practical Task implementations.

After that we can look at:

eventual support for pattern matching tasks?

We want something like datazen's TargetResolver class (and probably its parsing implementation).

[vkottler]

Because it can add a lot of complexity, we may want to start with the pattern matching

[vkottler]

Now that we have dynamic targets implemented, the last thing we need to test is that dynamic targets re-trigger each other.

When a Task executes, it probably needs to check its caller's outbox for dynamic target info.

[vkottler]

We should just test some more realistic scenarios and make sure we know what the desired behavior we want is.

[vkottler]

A task should be told at init time whether or not it's dynamic (e.g. provided by TaskManager since it makes the actual Target object).

If we're a dynamic task being executed and our private inbox doesn't have substitutions, we need to check the caller task's inbox potentially?

The problem here is that we want be able to get a correct literal value for ourselves, so we probably need to build one (e.g. take name and combine it with parent literal?)

[vkottler]

Also, if we want to follow that private inbox chain then we need to set our own private inbox parameters to our parent's inbox data as well

[vkottler]

In the set for literals we need to serialize substitutions (JSON?) and check for that as well, otherwise we would have to do some kind of literal name un-mangling to see if we should already be resolved

[vkottler]

We finally have an implementation for dynamic-target-based task dispatching (including dispatching dependencies correctly).

We need some more rigorous tests to ensure that the desired behavior is achieved (dynamic tasks triggering dynamic dependencies and literal dependencies successfully, dependencies invoked on their own later correctly resolved, etc.)

Once we're confident the basic task execution and dependency management works, we should implement a subprocess wrapper Task as an initial demonstration

[vkottler]

Implementation is done at this point, we need to clean up the documentation.

After this I think we should make some kind of interactive shell with this, could be interesting (e.g. like some program registers all of its dependencies, then you can execute them interactively

[vkottler]

Closing this as I'm about to merge it, time to use it for vmklib and maybe replace some datazen code