Script of Scripts (SoS) consists of
SoS Notebook, a Jupyter-based polyglot notebook that allows the use of multiple Jupyter kernels in one notebook, and
SoS Workflow, a workflow system for the execution of workflows in both process- and outcome-oriented styles. It is designed for data scientists and bioinformatics who routinely work with scripts in different languages such as bash, Python, R, and SAS. This repository contains the
SoS Workflow workflow engine.
Please refer to relevant publications SoS Notebook: An Interactive Multi-Language Data Analysis Environment in Bioinformatics, Script of Scripts: a pragmatic workflow system for daily computational research in PLoS Computational Biology, and the SoS homepage for more information on SoS Notebook and SoS Workflow.
We welcome and value community contributions: please post issues to provide us feedback or get our support; please send pull requests if you have helped fixing bugs or making improvements to the source code. In addition, if you are interesting in extending SoS, for example adding a new target type or task engine, please have a look at our tutorial on extending SoS here.
Installing SoS Workflow and Notebook
If you are using conda, you can install sos using command
conda install sos -c conda-forge
You can also install a majority of the SoS suite of tools using commands such as
conda install sos sos-pbs sos-notebook jupyterlab-sos sos-bash sos-python sos-r -c conda-forge
This will install SoS Workflow, SoS Notebook and its JupyterLab extension, language modules for
r-irkerneletc) and needed libraries (e.g.
feather) if needed.
If you are not using conda, you will have to install all the pieces one by one, but you will also have the freedom to use non-conda installation of
Juliaetc. With a working Python 3.6 installation, you can install SoS Workflow with command
pip install sos
pip install sos-pbs
if you would like to use SoS with a remote batch system such as LSF or Slurm.
You can install SoS Notebook, and register the sos kernel with Jupyter using the following commands
pip install sos-notebook python -m sos_notebook.install
To exchange data between live kernels in SoS Notebook, you will need to install individual kernels (e.g.
R), make sure they work under Jupyter, and install SoS language modules using commands such as
pip install sos-r sos-matlab sos-python sos-bash
Different modules might be needed to assist data exchange among kernels. Please refer to the installation page of sos website for details.
Change Log of SoS Workflow and SoS Notebook
- sos#1420: Allow checking the status, killing, and purge of remote workflows.
SoS Notebook 0.22.3
- sos-notebook#303: Make task execution non-blocking in sos notebook, so that the buttons to check status and remove tasks actually operatable.
SoS Notebook 0.22.2
- sos-notebook#307: Use a new template organization method for nbconvert > 0.6.0.
- sos#1392: Allow the use of
pem_fileto authenticate to remote hosts.
- sos#1319: Extend option
-r hostto allow workflow to be executed with templates and on PBS
- sos#1322: Deprecate option
-b BIN_DIRdue to sos#1319
- sos#1273: Allow workflows to be executed on cluster systems.
- sos#1277: Allow master tasks to be executed on cluster systems with extended runtime option
- sos#1279: Extend option
sos executeto use remote workers.
- sos#1288: Change the default value of option
- sos#1056, sos#1218: Use a new shared execution model to enforce optio
- sos#1219: Make sure global sections are executed only once
SoS Notebook 0.18.5
- sos-notebook#192: Allow language modules to support pattern matched kernel names such as julia-?.?
- sos#1206: Allow
output_from(step)to obtain output from a workflow.
- sos#1186: Add command line option
-Tto enfore dependency tracing.
- sos#1197: Introduce function
tracedto make dependencies that will always be traced
- sos#1201: Introduce dependencies between compounded workflows
- sos#1185: Disallow the use of parameters in tasks.
- sos#1186: (reverted) Enforcing the concept that targets in
dependsstatement always try to build dependency.
SoS Notebook 0.18.1
- sos-notebook#178: Allow adding
&to the end of magics
%runfileto execute workflow in background.
- sos-notebook#179: Remove magic
%rerunand add magic
- sos-notebook#180: Add option
%saveto execute the cell after saving it.
- sos#1115: Function
named_outputto support named inputs and outputs, among other new features summarized in this ticket.
- sos#1120: Allow paremeters
paired_withetc for functions
- sos#1125: Set
concurrent=Trueas default for substep execution.
- sos#1132: Deprecate action
stop_ifand replace it with
- sos#1175: Enforce the use of
sos_variableto import shared variable in a step
SoS Notebook 0.18.0
- sos-notebook#150: A new side panel that works in the same fashion as JupyterLab's console window.
- sos-notebook#154: New task status table that allows reporting status, killing, and purging multiple tasks with the same tags.
SoS 0.9.16.0, SoS Notebook 0.9.16.0
- sos#991: Use a new task file format to replace multiple files for each task. This change is not backward compatible so please upgrade only after you completed and removed all existing tasks.
- sos-notebook#89: Added templates to highlight source code using codemirror, with optional auto-generated table of contents.
- sos#983: Allow depending on an entire process-oriented workflow using a
sos_step()target that matches multiple steps.
- sos#975: Add option
-pto generate a summary report after the completion of workflow.
- sos#976: Much improved workflow help message (
sos run script -h).
SoS Notebook 0.9.14.4:
- sos-notebook#79: Allow auto-completion and inspection in subkernel cells.
SoS Notebook 0.9.14.1
- sos-notebook#74: Add a
%revisionsmagic to display revision history of the current document.
- sos#925: Output summary of executed and ignored step, substeps, and tasks after the execution of workflows.
SoS Notebook 0.9.13.4
- jupyterlab-sos#11: Magic
%cdnow changes directory of all subkernels
SoS Notebook 0.9.12.12
- sos-notebook#52: All new syntax highlighter that highlights expanded expressions
- sos-notebook#58: Stop removing leading comments from cells.
- sos#922: Use user-id for docker execution (-u)
- sos#926: Add function
zap()to SoS path classes
SoS Notebook 0.9.12.11
- sos-notebook#44: Allow sending text in markdown cells to side panel for execution.
- sos-notebook#47: Allow clear any HTML element with magic
- sos-notebook#50: Re-design logo for SoS Notebook.
- sos#914: Allow option
activeof actions and tasks to accept conditions.
- sos#915: Automatically expand user (
~) for SoS path types
- sos#916: Use hashlib instead of faster xxhash under windows
SoS Notebook 0.9.12.9
- sos-notebook#41: Stop saving unused kernels in sos notebook.
- sos#859: Introduce automatic auxiliary steps to simplify the use of makefile steps.
- sos#879: Add action options
stderrto redict output from script-executing actions.
- sos-notebook#42: Add option
- sos-notebook#39: Separation installation and deployment and use command
python -m sos_notebook.installto install
soskernel to Jupyter.
- sos#874: Add input option
concurrent=Trueto allow parallel execution of input groups.
- sos#874: Optimize task submission of task engines to reduce status checking
SoS Notebook 0.9.10.17
- sos-notebook#32: Add magic
%captureto capture output of cells