Execution and dispatching

[eirrgang]

This issue covers the infrastructure and protocol for updating the workflow during an scoped period of execution. Abstractly, a dispatching framework allows work to be executed locally or remotely in order to advance the state of the workflow in a given program scope.

Resolution of this issue may look like the NodeBuilder protocol described for gmxapi.
Some diagrams and discussion can be retrieved from past work to further document this issue.

Current works in progress can use this issue for tracking:

• Track current context and dispatch hierarchy
• Define dispatch protocol. e.g.
  1. make resources,
  2. apply resources to get callable / awaitable,
  3. finalize task,
  4. get and convey return object to parents or subscribers

Work on this issue has included introduction of scalems.execution and scalems.dispatching modules. Much of the execution module code for scalems.local and scalems.radical has been updated to just compose specific functions into core roles. Some details still need some clean up w.r.t. backend module configuration and specialized runtime implementation.

scalems.radical.runtime

• Combine Runtime and Configuration.
• Let Runtime exist without a Session.
• Introduce proxy object RuntimeState. Runtime resources are not accessible
  directly, but must be accessed through a RuntimeState, acquired through the
  context manager protocol or open() and close() protocol. RuntimeState
  inherits the Runtime interface for nesting.
• Runtime and RuntimeState must behave appropriately when errored or terminated
  unexpectedly in the context of a nested RuntimeState.
• Use Runtime instances in the pytest fixtures for more normative interaction.
• Allow RuntimeState to be provided to RuntimeManager or the WorkflowManager.dispatch() call that launches it.
• RuntimeState exposes weakref.WeakProxy handles to rp components to minimize
  chances of unexpected extension of reference lifetimes, and the RuntimeState
  object itself raises ScopeError if accessed after close() (such as by leaving the
  context manager).
• Add some locking for state changes until we are clearer about multithread use
  cases and safe state maintenance.
• Express and check for workflow software dependencies
• Uniquely name (and track) rp.Master task(s).
• Capture and process output from rp.Master task (whether failed or successful).

See also

Relates to #15

[eirrgang]

This issue is superseded in various facets by more recent issues.