papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks. It lets you parameterize and execute notebooks in batch mode.
SoS Notebook is a Jupyter kernel that allows the use of multiple kernels in one Jupyter notebook. Using language modules that understand datatypes of underlying languages (modules sos-bash, sos-r, sos-matlab, etc), SoS Notebook allows data exchange among live kernels of supported languages. SoS Notebook is also a frontend to the SoS Workflow that allows the development and execution of workflows from Jupyter notebooks.
Because the default papermill executor assumes a single kernel for the entire notebook, sos-papermill is provided as a customized engine for the execution of SoS notebooks.
pip install sos-papermill
or
conda install sos-papermill -c conda-forge
if you are using a conda environment.
Note that you will need to install sos-notebook, all relevant kernels (e.g. bash_kernel, irkernel) and related language modules (e.g. sos-bash, sos-r) to execute notebooks that use these kernels. Please refer to Running SoS for details on how to install SoS Notebook.
sos-papermill provides sos engine for papermill. All you need to do is to add option --engine sos to any papermill command that you might use:
papermill --engine sos [other options]
For example, to execute a parametrized notebook with parameter cutoff, you can use command
papermill --engine sos my_experiment.ipynb experiment_cutoff_2.ipynb -y '{"cutoff": 2}'
Please refer to the Papermill documentation for details on the use of papermill.
Note that parameters can be defined in either a SoS or a subkernel cell but in both cases
parameters should be passed in Python syntax. Parameters defined in a subkernel will be automatically
transferred to the subkernel using a %put magic.