Various improvements for Pipeline

examples/user_guide/Pipelines.ipynb

@@ -44,7 +44,9 @@
     "\n",
     "* ``param.output(c=param.Number, d=param.String)`` or ``param.output(('c', param.Number), ('d', param.String))``\n",
     "\n",
-    "In the example below the output is simply the result of multiplying the two inputs (``a`` and ``b``) which will produce output ``c``. Additionally we declare a ``view`` method which returns a ``LaTeX`` pane which will render the equation to ``LaTeX``. Finally a ``panel`` method declares returns a Panel object rendering both the parameters and the view; this is the second convention that a ``Pipeline`` expects.\n",
+    "In the example below the output is simply the result of multiplying the two inputs (``a`` and ``b``) which will produce output ``c``. Additionally we declare a ``view`` method which returns a ``LaTeX`` pane which will render the equation to ``LaTeX``. Finally a ``panel`` method should be implemented to return a Panel object providing a visual representation of the stage; this is the second convention that a ``Pipeline`` expects.\n",
+    "\n",
+    "In addition to the output parameters, as long as ``inherit_params`` is set any parameters which exist between two different stages will also be passed along.\n",
     "\n",
     "Let's start by displaying this stage on its own:"
    ]
@@ -56,24 +58,28 @@
    "outputs": [],
    "source": [
     "class Stage1(param.Parameterized):\n",
-    "    \n",
+    "\n",
     "    a = param.Number(default=5, bounds=(0, 10))\n",
     "\n",
     "    b = param.Number(default=5, bounds=(0, 10))\n",
     "    \n",
+    "    ready = param.Boolean(default=False, precedence=-1)\n",
+    "\n",
     "    @param.output(('c', param.Number), ('d', param.Number))\n",
     "    def output(self):\n",
     "        return self.a * self.b, self.a ** self.b\n",
     "    \n",
     "    @param.depends('a', 'b')\n",
     "    def view(self):\n",
+    "        if self.a > 5:\n",
+    "            self.ready = True\n",
     "        c, d = self.output()\n",
     "        return pn.pane.LaTeX('${a} * {b} = {c}$\\n${a}^{{{b}}} = {d}$'.format(\n",
     "            a=self.a, b=self.b, c=c, d=d), style={'font-size': '2em'})\n",
     "\n",
     "    def panel(self):\n",
     "        return pn.Row(self.param, self.view)\n",
-    "    \n",
+    "\n",
     "stage1 = Stage1()\n",
     "stage1.panel()"
    ]
@@ -84,9 +90,9 @@
    "source": [
     "To summarize we have followed several conventions when setting up this stage of our ``Pipeline``:\n",
     "\n",
-    "1. Declare a Parameterized class with some input parameters\n",
-    "2. Declare one or more output methods and name them appropriately\n",
-    "3. Declare a ``panel`` method that returns a view of the object that the ``Pipeline`` can render\n",
+    "1. Declare a Parameterized class with some input parameters.\n",
+    "2. Declare one or more methods decorated with the `param.output` decorator.\n",
+    "3. Declare a ``panel`` method that returns a view of the object that the ``Pipeline`` can render.\n",
     "\n",
     "Now that the object has been instantiated we can also query it for its outputs:"
    ]
@@ -116,6 +122,13 @@
     "pipeline.add_stage('Stage 1', stage1)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The ``add_stage`` method takes the name of the stage as its first argument, the stage class or instance as the second parameter and any additional keyword arguments to override default behavior."
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -178,6 +191,9 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "pipeline = pn.pipeline.Pipeline(debug=True)\n",
+    "pipeline.add_stage('Stage 1', Stage1(), ready_parameter='ready', auto_advance=True)\n",
+    "pipeline.add_stage('Stage 2', Stage2)\n",
     "pipeline.layout"
    ]
   },
@@ -211,6 +227,231 @@
    "source": [
     "As you will note the Pipeline stages may be either ``Parameterized`` instances or ``Parameterized`` classes, but when working with instances you must ensure that updating the parameters of the class is sufficient to update the current state of the class."
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Non-linear pipelines\n",
+    "\n",
+    "Pipelines are not limited to simple linear UI workflows, they support any arbitrary branching structures, or put another way any acyclic graph. A simple example might be a workflow with two alternative stages which rejoin at a later point. In the very simple example below we declare four stages, an `Input`, `Multiply` and `Add` and finally a `Result` stage."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class Input(param.Parameterized):\n",
+    "    \n",
+    "    value1 = param.Integer(default=0)\n",
+    "\n",
+    "    value2 = param.Integer(default=0)\n",
+    "    \n",
+    "    def panel(self):\n",
+    "        return pn.Row(self.param.value1, self.param.value2)\n",
+    "\n",
+    "class Multiply(Input):\n",
+    "    \n",
+    "    def panel(self):\n",
+    "        return '%.3f * %.3f' % (self.value1, self.value2)\n",
+    "\n",
+    "    @param.output('result')\n",
+    "    def output(self):\n",
+    "        return self.value1 * self.value2\n",
+    "    \n",
+    "class Add(Input):\n",
+    "    \n",
+    "    def panel(self):\n",
+    "        return '%d + %d' % (self.value1, self.value2)\n",
+    "    \n",
+    "    @param.output('result')\n",
+    "    def output(self):\n",
+    "        return self.value1 + self.value2\n",
+    "    \n",
+    "class Result(Input):\n",
+    "    \n",
+    "    result = param.Number(default=0)\n",
+    "\n",
+    "    def panel(self):\n",
+    "        return self.result\n",
+    "    \n",
+    "dag = pn.pipeline.Pipeline()\n",
+    "\n",
+    "dag.add_stage('Input', Input)\n",
+    "dag.add_stage('Multiply', Multiply)\n",
+    "dag.add_stage('Add', Add)\n",
+    "dag.add_stage('Result', Result)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "After adding all the stages we have to express the relationship between these stages. To declare the graph we can use the ``define_graph`` method and provide a adjacency map, which declares which stage feeds into which other stages. In this case the `Input` feeds into both ``Multiply`` and ``Add`` and both those stages feed into the ``Result``:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dag.define_graph({'Input': ('Multiply', 'Add'), 'Multiply': 'Result', 'Add': 'Result'})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is of course a very simple example but it demonstrates the ability to express arbitrary workflows with branching and converging steps:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dag.layout"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Custom layout\n",
+    "\n",
+    "The `Pipeline` has a default `layout` with a `header` bar containing the navigation elements and the main `stage`. To achieve more custom layouts, each of the components can be arranged manually:\n",
+    "\n",
+    "* `layout`: The overall layout of the header and stage.\n",
+    "* `header`: The navigation components and network diagram.\n",
+    "  - `title`: The name of the current stage.\n",
+    "  - `network`: A network diagram representing the pipeline.\n",
+    "  - `buttons`: All navigation buttons and selectors.\n",
+    "  - `prev_button`: The button to go to the previous stage.\n",
+    "  - `prev_selector`: The selector widget to select between previous branching stages.\n",
+    "  - `next_button`:   The button to go to the previous stage\n",
+    "  - `next_selector`: The selector widget to select the next branching stages.\n",
+    "* `stage`: The contents of the current pipeline stage.\n",
+    "\n",
+    "When displaying a pipeline in this way it is important to first initialize it:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dag.init()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now we can compose the components in any way that is desired:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pn.Column(\n",
+    "    pn.Row(dag.title, pn.layout.HSpacer(), dag.buttons),\n",
+    "    dag.network,\n",
+    "    dag.stage\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Programmatic flow control\n",
+    "\n",
+    "By default controlling the flow between different stages is done using the \"Previous\" and \"Next\" buttons. However often we want to control the UI flow programmatically from within a stage. A `Pipeline` allows programmatic control by declaring a `ready_param` either per stage or globally on the Pipeline, which blocks and unblocks the buttons and allow advancing automatically when combined with the ``auto_advance`` parameter. In this way we can control the workflow programmatically from inside the stages.\n",
+    "\n",
+    "In the example below we create a version of the previous workflow which can be used without the buttons by declaring ``ready`` parameters on each of the stages, which we can toggle with a custom button or simply set to `True` by default to automatically skip the stage."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "class AutoInput(Input):\n",
+    "\n",
+    "    operator = param.Selector(default='+', objects=['*', '+'])\n",
+    "    \n",
+    "    ready = param.Boolean(default=False)\n",
+    "    \n",
+    "    def panel(self):\n",
+    "        button = pn.widgets.Button(name='Go', button_type='success')\n",
+    "        button.on_click(lambda event: setattr(self, 'ready', True))\n",
+    "        return pn.Column(\n",
+    "            pn.Row(self.param.value1, self.param.operator, self.param.value2),\n",
+    "            button\n",
+    "        )\n",
+    "\n",
+    "class AutoMultiply(Multiply):\n",
+    "    \n",
+    "    ready = param.Boolean(default=True)\n",
+    "\n",
+    "class AutoAdd(Add):\n",
+    "    \n",
+    "    ready = param.Boolean(default=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now that we have declared these stages let us set up the pipeline ensuring that we declare the `ready_parameter` and `auto_advance` settings:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dag = pn.pipeline.Pipeline() # could set ready_parameter='ready' and auto_advance=True globally\n",
+    "\n",
+    "dag.add_stage('Input', AutoInput, ready_parameter='ready', auto_advance=True)\n",
+    "dag.add_stage('Multiply', AutoMultiply, ready_parameter='ready', auto_advance=True)\n",
+    "dag.add_stage('Add', AutoAdd, ready_parameter='ready', auto_advance=True)\n",
+    "dag.add_stage('Result', Result)\n",
+    "\n",
+    "dag.define_graph({'Input': ('Multiply', 'Add'), 'Multiply': 'Result', 'Add': 'Result'})\n",
+    "\n",
+    "dag.init()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Finally we display the pipeline, without the buttons because we don't need them:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pn.Column(\n",
+    "    dag.title,\n",
+    "    dag.network,\n",
+    "    dag.stage\n",
+    ")"
+   ]
   }
  ],
  "metadata": {
@@ -220,5 +461,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
 }