guides/cron/README.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca934856",
   "metadata": {
    "papermill": {
     "duration": 0.049223,
     "end_time": "2022-03-12T19:09:02.782991",
     "exception": false,
     "start_time": "2022-03-12T19:09:02.733768",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "To run this locally, [install Ploomber](https://docs.ploomber.io/en/latest/get-started/quick-start.html) and execute: `ploomber examples -n guides/cron`\n",
    "\n",
    "Found an issue? [Let us know.](https://github.com/ploomber/projects/issues/new?title=guides/cron%20issue)\n",
    "\n",
    "Questions? [Ask us on Slack.](https://ploomber.io/community/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e13e3b1",
   "metadata": {
    "papermill": {
     "duration": 0.041039,
     "end_time": "2022-03-12T19:09:02.862739",
     "exception": false,
     "start_time": "2022-03-12T19:09:02.821700",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Scheduling with cron\n",
    "\n",
    "This guide shows how to schedule Ploomber pipelines using `cron`.\n",
    "\n",
    "**Note:** `cron` is only available on macOS and Linux.\n",
    "\n",
    "## Pre-requisites\n",
    "\n",
    "Ensure cron is installed:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc2f1cab",
   "metadata": {
    "papermill": {
     "duration": 0.037729,
     "end_time": "2022-03-12T19:09:02.940515",
     "exception": false,
     "start_time": "2022-03-12T19:09:02.902786",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```sh\n",
    "crontab -l\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6cac96d",
   "metadata": {
    "papermill": {
     "duration": 0.045923,
     "end_time": "2022-03-12T19:09:03.026574",
     "exception": false,
     "start_time": "2022-03-12T19:09:02.980651",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "If you don't see a \"command not found error\", you have `cron` installed and can continue.\n",
    "\n",
    "## Get the example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12bed96a",
   "metadata": {
    "papermill": {
     "duration": 0.039559,
     "end_time": "2022-03-12T19:09:03.106888",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.067329",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```sh\n",
    "pip install ploomber\n",
    "ploomber examples -n guides/cron -o cron\n",
    "cd cron\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "567069ce",
   "metadata": {
    "papermill": {
     "duration": 0.044075,
     "end_time": "2022-03-12T19:09:03.191974",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.147899",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Setup\n",
    "\n",
    "Configure virtual environment:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d2faa97",
   "metadata": {
    "papermill": {
     "duration": 0.038337,
     "end_time": "2022-03-12T19:09:03.270925",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.232588",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```sh\n",
    "ploomber install --create-env --use-venv\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece78ab8",
   "metadata": {
    "papermill": {
     "duration": 0.04095,
     "end_time": "2022-03-12T19:09:03.352933",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.311983",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Activate environment:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6f77aea",
   "metadata": {
    "papermill": {
     "duration": 0.038889,
     "end_time": "2022-03-12T19:09:03.429870",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.390981",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```sh\n",
    "source venv-cron/bin/activate\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56ebdfe3",
   "metadata": {
    "papermill": {
     "duration": 0.043446,
     "end_time": "2022-03-12T19:09:03.512014",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.468568",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## The code\n",
    "\n",
    "The pipeline has two stages: load, and plot:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8489e05",
   "metadata": {
    "lines_to_next_cell": 2,
    "papermill": {
     "duration": 0.040839,
     "end_time": "2022-03-12T19:09:03.592240",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.551401",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```yaml\n",
    "# Content of pipeline.yaml\n",
    "tasks:\n",
    "  - source: scripts/load.py\n",
    "    product:\n",
    "      nb: products/{{timestamp}}/load.html\n",
    "      data: products/{{timestamp}}/load.csv\n",
    "\n",
    "  - source: scripts/plot.py\n",
    "    product:\n",
    "      nb: products/{{timestamp}}/plot.html\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e5a1224",
   "metadata": {
    "papermill": {
     "duration": 0.040496,
     "end_time": "2022-03-12T19:09:03.671763",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.631267",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Notice that all the products are parametrized prefixed by: `products/{{timestamp}}`; this will allow us to store the products of each run depending on the runtime timestamp.\n",
    "\n",
    "## Development\n",
    "\n",
    "When modifying the pipeline, you can call the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b2460656",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-12T19:09:03.760960Z",
     "iopub.status.busy": "2022-03-12T19:09:03.760251Z",
     "iopub.status.idle": "2022-03-12T19:09:13.642901Z",
     "shell.execute_reply": "2022-03-12T19:09:13.643445Z"
    },
    "papermill": {
     "duration": 9.933065,
     "end_time": "2022-03-12T19:09:13.643821",
     "exception": false,
     "start_time": "2022-03-12T19:09:03.710756",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading pipeline...\n",
      "name    Ran?      Elapsed (s)    Percentage\n",
      "------  ------  -------------  ------------\n",
      "load    True          1.90359       39.4266\n",
      "plot    True          2.9246        60.5734\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Edu/miniconda3/envs/projects/lib/python3.9/site-packages/papermill/iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.\n",
      "  from pyarrow import HadoopFileSystem\n",
      "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:62: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
      "  warnings.warn(res)\n",
      "Building task 'load':   0%|          | 0/2 [00:00<?, ?it/s]\n",
      "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
      "Executing: 100%|██████████| 4/4 [00:01<00:00,  2.67cell/s]\u001b[A\n",
      "Building task 'plot':  50%|█████     | 1/2 [00:01<00:01,  1.91s/it]\n",
      "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
      "Executing:  25%|██▌       | 1/4 [00:02<00:06,  2.10s/cell]\u001b[A\n",
      "Executing: 100%|██████████| 4/4 [00:02<00:00,  1.54cell/s]\u001b[A\n",
      "Building task 'plot': 100%|██████████| 2/2 [00:04<00:00,  2.42s/it]\n",
      "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:98: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
      "  warnings.warn(res)\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "ploomber build"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf0a6f1",
   "metadata": {
    "papermill": {
     "duration": 0.045847,
     "end_time": "2022-03-12T19:09:13.735306",
     "exception": false,
     "start_time": "2022-03-12T19:09:13.689459",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Note that products will go to `output/dev/`.\n",
    "\n",
    "For scheduling the workflow, we need to tell Ploomber to use a different configuration file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "07580831",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-12T19:09:13.826630Z",
     "iopub.status.busy": "2022-03-12T19:09:13.825907Z",
     "iopub.status.idle": "2022-03-12T19:09:23.486023Z",
     "shell.execute_reply": "2022-03-12T19:09:23.486932Z"
    },
    "papermill": {
     "duration": 9.709503,
     "end_time": "2022-03-12T19:09:23.487151",
     "exception": false,
     "start_time": "2022-03-12T19:09:13.777648",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading pipeline...\n",
      "name    Ran?      Elapsed (s)    Percentage\n",
      "------  ------  -------------  ------------\n",
      "load    True          1.99751       40.9529\n",
      "plot    True          2.88008       59.0471\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Edu/miniconda3/envs/projects/lib/python3.9/site-packages/papermill/iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.\n",
      "  from pyarrow import HadoopFileSystem\n",
      "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:62: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
      "  warnings.warn(res)\n",
      "Building task 'load':   0%|          | 0/2 [00:00<?, ?it/s]\n",
      "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
      "Executing: 100%|██████████| 4/4 [00:01<00:00,  2.48cell/s]\u001b[A\n",
      "Building task 'plot':  50%|█████     | 1/2 [00:02<00:02,  2.00s/it]\n",
      "Executing:   0%|          | 0/4 [00:00<?, ?cell/s]\u001b[A\n",
      "Executing:  25%|██▌       | 1/4 [00:01<00:05,  1.80s/cell]\u001b[A\n",
      "Executing: 100%|██████████| 4/4 [00:02<00:00,  1.58cell/s]\u001b[A\n",
      "Building task 'plot': 100%|██████████| 2/2 [00:04<00:00,  2.44s/it]\n",
      "/Users/Edu/dev/ploomber/src/ploomber/cli/build.py:98: UserWarning: b'Error: Non cloud users can monitor up to 2 pipelines simultaneously'\n",
      "  warnings.warn(res)\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "# tell ploomber to use env.cron.yaml as config file\n",
    "export PLOOMBER_ENV_FILENAME=env.cron.yaml\n",
    "# build pipeline\n",
    "ploomber build\n",
    "# delete env variable\n",
    "unset PLOOMBER_ENV_FILENAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02554a25",
   "metadata": {
    "papermill": {
     "duration": 0.043817,
     "end_time": "2022-03-12T19:09:23.577101",
     "exception": false,
     "start_time": "2022-03-12T19:09:23.533284",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Let's see the contents of the products directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e4da0559",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-03-12T19:09:23.668551Z",
     "iopub.status.busy": "2022-03-12T19:09:23.667747Z",
     "iopub.status.idle": "2022-03-12T19:09:23.730703Z",
     "shell.execute_reply": "2022-03-12T19:09:23.731378Z"
    },
    "papermill": {
     "duration": 0.111568,
     "end_time": "2022-03-12T19:09:23.731579",
     "exception": false,
     "start_time": "2022-03-12T19:09:23.620011",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-03-12T13:09:16.783678\n",
      "dev\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "ls products"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e440e91",
   "metadata": {
    "papermill": {
     "duration": 0.043804,
     "end_time": "2022-03-12T19:09:23.822190",
     "exception": false,
     "start_time": "2022-03-12T19:09:23.778386",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You should see two folders, `dev/` and another one with the runtime timestamp."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53df9d09",
   "metadata": {
    "papermill": {
     "duration": 0.0453,
     "end_time": "2022-03-12T19:09:23.911172",
     "exception": false,
     "start_time": "2022-03-12T19:09:23.865872",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Scheduling\n",
    "\n",
    "Now let's schedule in cron. First, to edit the cron configuration file:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34bb154",
   "metadata": {
    "papermill": {
     "duration": 0.043471,
     "end_time": "2022-03-12T19:09:23.999080",
     "exception": false,
     "start_time": "2022-03-12T19:09:23.955609",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```sh\n",
    "crontab -e\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "778e13fa",
   "metadata": {
    "papermill": {
     "duration": 0.044044,
     "end_time": "2022-03-12T19:09:24.091740",
     "exception": false,
     "start_time": "2022-03-12T19:09:24.047696",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Note that this will open the configuration file in the default editor,\n",
    "if you don't know what that is, you can open it with `nano`:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "774ff38f",
   "metadata": {
    "papermill": {
     "duration": 0.045209,
     "end_time": "2022-03-12T19:09:24.179515",
     "exception": false,
     "start_time": "2022-03-12T19:09:24.134306",
     "status": "completed"
    },
    "region_name": "md",
    "tags": []
   },
   "source": [
    "```sh\n",
    "EDITOR=nano crontab -e\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7d8add9",
   "metadata": {
    "papermill": {
     "duration": 0.04465,
     "end_time": "2022-03-12T19:09:24.269856",
     "exception": false,
     "start_time": "2022-03-12T19:09:24.225206",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Once the editor opens, add a line like this:\n",
    "\n",
    "```txt\n",
    "* * * * *  PROJ=/path/to/cron && cd $PROJ && bash run.sh >> cron.log 2>&1\n",
    "```\n",
    "\n",
    "**Note:** If using macOS Big Sur (11.6) or newer, you may need to follow a few [extra steps](https://osxdaily.com/2020/04/27/fix-cron-permissions-macos-full-disk-access/) to enable cron."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b1878ae",
   "metadata": {
    "papermill": {
     "duration": 0.043993,
     "end_time": "2022-03-12T19:09:24.362439",
     "exception": false,
     "start_time": "2022-03-12T19:09:24.318446",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Replace the `/path/to/cron/` with the absolute path to the `cron/` directory that contains the sample code. (Tip: so get the absolute path, enter `pwd` in your terminal).\n",
    "\n",
    "If you opened the configuration file with `nano`, save your changes with `CTRL + O` and exit the editor with `CTRL + X`.\n",
    "\n",
    "After a minute, you'll start to see more directories in the products folder; this is what mine looks like:\n",
    "\n",
    "```\n",
    "2022-03-12T11:14:47.506532/ \n",
    "2022-03-12T11:25:12.707618/ \n",
    "dev/\n",
    "```\n",
    "\n",
    "If you see something like this, congratulations, you have a scheduled pipeline up and running!\n",
    "\n",
    "To learn how to modify the scheduling interval, see the Overview section in cron's [Wikipedia article.](https://en.wikipedia.org/wiki/Cron)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d665ad1f",
   "metadata": {
    "papermill": {
     "duration": 0.04505,
     "end_time": "2022-03-12T19:09:24.450267",
     "exception": false,
     "start_time": "2022-03-12T19:09:24.405217",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Troubleshooting\n",
    "\n",
    "If you don't see the new directories, check out the `cron.log` file, which will contain any error messages, and ping us [on Slack](https://ploomber.io/community) so we can help you."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 23.534977,
   "end_time": "2022-03-12T19:09:24.709663",
   "environment_variables": {},
   "exception": null,
   "input_path": "guides/cron/_build/readme_preprocessed.ipynb",
   "output_path": "guides/cron/README.ipynb",
   "parameters": {},
   "start_time": "2022-03-12T19:09:01.174686",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}