support async Python actions

[wahuneke]

Motivations

Two motivations for this (sizeable) request:

• it may provide a simpler route to parallel action support (which is currently unsupported in Windows and has limitations elsewhere)
• allow for integration against async libraries (e.g. aiodocker)

Usage

1. Declaring a Python async function as a task's action would result in that task's action being pushed onto the event loop. Additional plumbing would be added to the loop as well in order to chain dependent tasks in downstream
2. Activating a global configuration option would cause doit to automatically run all shell command blocks using an async shell exec adapter such as Python's async subprocess feature

The work

I have not looked at doit code yet, but would consider volunteering for some / all of this effort if more-experienced contributors believe it is worth pursuing.

I did some quick searches through PRs and existing issues and did not see any discussion of Python's async faculties... Sorry if I missed something or if this is already asked and answered

[schettino72]

This would definitely be a nice addition.

The code you are looking for is at runner.py.
You can see default implementation and parallel multiprocess/thread.

Actually I have done some experimentation related to this a while ago (in a different project)...
I want to move away from using the multiprocess module altogether.

I guess a first step would be to just support async actions (run just one at time).

As of today you can specify the parallel mechanism only per run (i.e. all tasks a threads or process).
Ideally would be able to group tasks and run them on different parallel strategies with master as async code.

See also

• Prohibit concurrent execution of tasks using shared resource #130
• Option/default to synchronize output printing with task when executing in parallel #236

[wahuneke]

Thank you for the pointers. I have looked over runner.py and can see that this may be the place to start. I also like the suggestion of starting with async actions but enforcing 'one at a time' at the task level.

I'm currently working on another doit extension... I'll return to this async idea after I get the alpha release for the other project :)