Using a sos namespace

[BoPeng]

One of the problems with sos if that sos is built on top of python so there is a high probability of name clash. For example, users could accidentally define run and clash with our run action.

We have tried to resolve this issue with _input, SOS_CONFIG etc, but things like python still has a high risk of name conflict. I propose that we put everything into a sos namespace so that

• sos.inputs is _input, the input for the current substep, which is the sos_target object.
• sos.input is _input[0], which is the default for input with a single input. No automatic _input to _input[0] will be performed (which complicated the code quite a bit).
• sos.group and sos.groups for results of group_by. In this way, group_by does not change _input by default.
• sos.outputs and sos.output follows the same convension.
• sos.step_input and sos.step_output are step_input and step_output.
• sos.config is the SOS_CONFIG
• sos.run_script("R", ...) is what R: will be translated to. The same goes to sos.python etc.. We may stick to sos.Retc, butsos.run` conflicts with the workflow execution function below.
• sos.run will be sos_run for executing subworkflows.
• sos.__dict__ will be sos_dict which is the namespace we use to store all runtime dictionaries.

In this way, sos adds on top of Python

1. Directives such as input:, output:, depends:, task:.
2. A syntax sugar for scripting running actions such as R: and Python:.
3. A sos namespace with data and functions.

Backward compatibility can be temporarily retained by making existing objects available (e.g. R:) but gives a deprecation warning.

Note that

1. the level of change may trigger the development of sos.2 in a separate repository, where we will reexamine the generic design and implementation issues with sos, such as remote/multi-node execution, and subworkflows.
2. we need to rigorously define these variables, how they are transferred (across steps, processes, and nodes), and how they change during the execution of the workflows. This is especially important for concurrent execution of substeps using async.