doc/source/developer/patterns/result_dependent_loop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a393bc36",
   "metadata": {},
   "source": [
    "# Result-Dependent Loops\n",
    "\n",
    "To iterate over the result from an electron, put the iteration (loop) logic inside another electron.\n",
    "\n",
    "## Context\n",
    "\n",
    "Often the output of one task is a collection that you want to iterate over using another task. In Covalent terms, this means you want to use an `electron` to produce an `iterable`, then process the iterable with another electron. In these cases, perform the iteration inside an electron. \n",
    "\n",
    "When a lattice is dispatched, the Covalent server executes the lattice in order to build the transport graph. The transport graph is then analyzed to parallelize the execution of electrons on their assigned executors.\n",
    "\n",
    "If the server encounters a loop over the output of an electron, it cannot infer the structure on which the loop depends (the size and composition of the iterable) and is prevented from building the transport graph.\n",
    "\n",
    "Putting the loop inside an electron defers resolution of the loop to when the electron is dispatched, and ensures that it takes place on the electron's executor. \n",
    "\n",
    "Note: This pattern applies only when the iterator is produced by an electron. Iterating on fixed values in a lattice as described [here](https://covalent.readthedocs.io/en/latest/how_to/coding/looping.html) does not require electron execution to evaluate the iterator and build the graph.\n",
    "\n",
    "## Best Practice\n",
    "\n",
    "Compute dynamically generated iterators inside an electron. Electrons' execution is deferred during the graph build phase, so their output cannot be used to build the transport graph and analyze the execution for parallelization. Instead, the electron is added to the transport graph and the loop is computed within the electron when it is executed.\n",
    "\n",
    "For result-dependent computations that might be too complex to encapsulate in a single electron, use a [sublattice](./dynamic_workflow.ipynb).\n",
    "\n",
    "## Example\n",
    "\n",
    "Contrast the two examples below.\n",
    "\n",
    "### Example 1: Incorrect\n",
    "\n",
    "This example demonstrates the incorrect approach: looping over a computed iterator in the lattice but not within an electron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "93409de5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status:  FAILED \n",
      " The following tasks failed:\n",
      "16: :task()[5]\n",
      "19: :task()[6]\n",
      "22: :task()[7]\n",
      "25: :task()[8]\n",
      "28: :task()[9]\n",
      "31: :task()[10]\n",
      "34: :task()[11]\n",
      "37: :task()[12]\n",
      "40: :task()[13]\n",
      "43: :task()[14]\n",
      "46: :task()[15]\n",
      "49: :task()[16]\n",
      "52: :task()[17]\n"
     ]
    }
   ],
   "source": [
    "import covalent as ct\n",
    "import random\n",
    "\n",
    "# Technique 1: Fails because the transport graph cannot be defined\n",
    "\n",
    "@ct.electron\n",
    "def task():\n",
    "    return random.sample(range(10, 30), 5)\n",
    "\n",
    "@ct.electron\n",
    "def task_2(x):\n",
    "    return x ** 2\n",
    "\n",
    "@ct.lattice\n",
    "def workflow_1():\n",
    "    random_list = task()\n",
    "\n",
    "    res = []\n",
    "    for rn in random_list:\n",
    "        res.append(task_2(rn))\n",
    "    \n",
    "    return rn\n",
    "\n",
    "id = ct.dispatch(workflow_1)()\n",
    "res = ct.get_result(id, wait=True)\n",
    "print(\"Status: \", res.status, \"\\n\", res.error) # (Selected output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b89859a8",
   "metadata": {},
   "source": [
    "### Example 2: Improved\n",
    "\n",
    "The iterator is passed to the second electron, which loops over it internally and returns the results in a list. In this case the loop is executed entirely at electron execution time, in the electron's executor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d727689b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Lattice Result\n",
      "==============\n",
      "status: COMPLETED\n",
      "result: [256, 100, 625, 196, 676]\n",
      "input args: []\n",
      "input kwargs: {}\n",
      "error: None\n",
      "\n",
      "start_time: 2023-03-13 20:26:42.797010\n",
      "end_time: 2023-03-13 20:26:42.985698\n",
      "\n",
      "results_dir: /Users/dave/.local/share/covalent/data\n",
      "dispatch_id: a03a79f3-748f-488d-8a7e-aebe78a4a26e\n",
      "\n",
      "Node Outputs\n",
      "------------\n",
      "task_1(0): [16, 10, 25, 14, 26]\n",
      "task_2_new(1): [256, 100, 625, 196, 676]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import covalent as ct\n",
    "import random\n",
    "\n",
    "# Technique 2: Iterator is contained in an electron\n",
    "\n",
    "@ct.electron\n",
    "def task_1():\n",
    "    return random.sample(range(10, 30), 5)\n",
    "\n",
    "# Method (2):\n",
    "@ct.electron\n",
    "def task_2_new(x_list):\n",
    "\n",
    "    squares = []\n",
    "    for x in x_list:\n",
    "        squares.append(x ** 2)\n",
    "    \n",
    "    return squares\n",
    "\n",
    "@ct.lattice\n",
    "def workflow_2():\n",
    "    random_list = task_1()\n",
    "    return task_2_new(random_list)\n",
    "\n",
    "id = ct.dispatch(workflow_2)()\n",
    "res = ct.get_result(id, wait=True)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6044a9ff",
   "metadata": {},
   "source": [
    "## See Also\n",
    "\n",
    "[Result-Dependent If-Else](./result_dependent_if_else.ipynb)\n",
    "\n",
    "[Dynamic Workflows](./dynamic_workflow.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}