examples/loremipsum/index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Example: Character frequencies in the Loremipsum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to setup and run this example:\n",
    "\n",
    "1. [Launch notebook on Binder](https://mybinder.org/v2/gh/riga/law/master?filepath=examples%2Floremipsum%2Findex.ipynb)\n",
    "2. [Static notebook on GitHub](https://github.com/riga/law/blob/master/examples/loremipsum/index.ipynb)\n",
    "3. Docker: `docker run -ti riga/law:example loremipsum`\n",
    "4. Local: `source setup.sh`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction\n",
    "\n",
    "This example demonstrates how to create and run a simple law task tree.\n",
    "\n",
    "The actual payload of the tasks is rather trivial. Six different versions of the [lorem ipsum](https://www.lipsum.com) dummy text are fetched from a website. Per version, the character frequencies are measured, and merged and visualized in the end.\n",
    "\n",
    "**You might want to checkout the implemention of the tasks below in ([tasks.py](https://github.com/riga/law/blob/master/examples/loremipsum/tasks.py)) while executing the notebook.**\n",
    "\n",
    "Resources: [luigi](http://luigi.readthedocs.io/en/stable), [law](http://law.readthedocs.io/en/latest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Before you proceed**, load the law ipython magics:\n",
    "\n",
    "- `%law`: runs the passed line in a subprocess\n",
    "- `%ilaw`: runs the passed line interactively in the current process (for tasks defined in notebooks)\n",
    "\n",
    "Since we do not define any tasks in this notebook, we are fine with `%law`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mlaw.contrib.ipython.magic\u001b[0m - \u001b[0;49;39mrunning initialization command 'source setup.sh'\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mlaw.contrib.ipython.magic\u001b[0m - \u001b[0;49;39mmagics successfully registered: %law, %ilaw\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import law\n",
    "law.contrib.load(\"ipython\")\n",
    "law.ipython.register_magics(init_cmd=\"source setup.sh\", line_cmd=\"source setup.sh\", log_level=\"INFO\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not specific to law but helps setting up the dependencies (luigi and six) in the example directory of this notebook.\n",
    "\n",
    "This is equivalent to `source setup.sh` when running the commands in a terminal.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Let law index your the tasks and their parameters (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that indexing is only required for auto-completion in the command line and therefore not that important for this notebook. However, it is a convenient feature to show your available tasks and complete their parameters when working with a terminal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "indexing tasks in 1 module(s)\r\n",
      "loading module 'tasks', \u001b[1;49;39mdone\u001b[0m\r\n",
      "\r\n",
      "module '\u001b[1;49;39mtasks\u001b[0m', 4 task(s):\r\n",
      "    - \u001b[0;49;32mShowFrequencies\u001b[0m\r\n",
      "    - \u001b[0;49;32mFetchLoremIpsum\u001b[0m\r\n",
      "    - \u001b[0;49;32mCountChars\u001b[0m\r\n",
      "    - \u001b[0;49;32mMergeCounts\u001b[0m\r\n",
      "\r\n",
      "written 4 task(s) to index file '/law/examples/loremipsum/.law/index'\r\n"
     ]
    }
   ],
   "source": [
    "%law index --verbose"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides, while *indexing* always sounds cumbersome, the law index file is just a human-readable file summarizing your tasks, the corresponding python modules, and their parameters. Have a look at the index file if you're interested. Note that the output of the cell below might be hidden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tasks:ShowFrequencies:fetch-output log-file print-deps print-output print-status remove-output slow\r\n",
      "tasks:FetchLoremIpsum:fetch-output file-index log-file print-deps print-output print-status remove-output slow\r\n",
      "tasks:CountChars:fetch-output file-index log-file print-deps print-output print-status remove-output slow\r\n",
      "tasks:MergeCounts:fetch-output log-file print-deps print-output print-status remove-output slow\r\n",
      "\r\n"
     ]
    }
   ],
   "source": [
    "%law index --show"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Check the status of the ShowFrequencies task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to use the `law run` command for the first time. But to begin with, we add a parameter `--print-status -1` to the command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print task status with max_depth -1 and target_depth 0\r\n",
      "\r\n",
      "0 > \u001b[0;49;32mShowFrequencies\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "│\r\n",
      "└──1 > \u001b[0;49;32mMergeCounts\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_merged.json)\r\n",
      "   │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_1.json)\r\n",
      "   │  │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_1.txt)\r\n",
      "   │             \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_2.json)\r\n",
      "   │  │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_2.txt)\r\n",
      "   │             \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_3.json)\r\n",
      "   │  │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_3.txt)\r\n",
      "   │             \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_4.json)\r\n",
      "   │  │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_4.txt)\r\n",
      "   │             \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_5.json)\r\n",
      "   │  │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_5.txt)\r\n",
      "   │             \u001b[1;49;31mabsent\u001b[0m\r\n",
      "   │\r\n",
      "   └──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "      │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_6.json)\r\n",
      "      │       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "      │\r\n",
      "      └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "               \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_6.txt)\r\n",
      "                 \u001b[1;49;31mabsent\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "%law run ShowFrequencies --print-status -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see that all output targets are absent and no task is complete yet.\n",
    "\n",
    "Although `law run` was called, no task was actually executed. A few parameters will make law to only print helpful information and then terminate, such as the status of a certain task (`ShowFrequencies` above) and its **recursive** dependencies. The value given to `--print-status` defines the recursion level, where `0` is the task given to `law run` itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print task status with max_depth 0 and target_depth 0\r\n",
      "\r\n",
      "0 > \u001b[0;49;32mShowFrequencies\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n"
     ]
    }
   ],
   "source": [
    "%law run ShowFrequencies --print-status 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other so-called *interactive* parameters are `--print-deps`, `--print-output`, `--fetch-output` and `--remove-output`. Use the help to find out more about these parameters. Note that the output of the cell below might be hidden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: law run [--local-scheduler [CORE_LOCAL_SCHEDULER]]\r\n",
      "               [--module CORE_MODULE] [--help [CORE_HELP]]\r\n",
      "               [--help-all [CORE_HELP_ALL]]\r\n",
      "               [--ShowFrequencies-log-file SHOWFREQUENCIES_LOG_FILE]\r\n",
      "               [--log-file LOG_FILE]\r\n",
      "               [--ShowFrequencies-print-deps SHOWFREQUENCIES_PRINT_DEPS]\r\n",
      "               [--print-deps PRINT_DEPS]\r\n",
      "               [--ShowFrequencies-print-status SHOWFREQUENCIES_PRINT_STATUS]\r\n",
      "               [--print-status PRINT_STATUS]\r\n",
      "               [--ShowFrequencies-print-output SHOWFREQUENCIES_PRINT_OUTPUT]\r\n",
      "               [--print-output PRINT_OUTPUT]\r\n",
      "               [--ShowFrequencies-remove-output SHOWFREQUENCIES_REMOVE_OUTPUT]\r\n",
      "               [--remove-output REMOVE_OUTPUT]\r\n",
      "               [--ShowFrequencies-fetch-output SHOWFREQUENCIES_FETCH_OUTPUT]\r\n",
      "               [--fetch-output FETCH_OUTPUT]\r\n",
      "               [--ShowFrequencies-slow [SHOWFREQUENCIES_SLOW]] [--slow [SLOW]]\r\n",
      "               [Required root task]\r\n",
      "\r\n",
      "positional arguments:\r\n",
      "  Required root task    Task family to run. Is not optional.\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  --local-scheduler [CORE_LOCAL_SCHEDULER]\r\n",
      "                        Use an in-memory central scheduler. Useful for\r\n",
      "                        testing.\r\n",
      "  --module CORE_MODULE  Used for dynamic loading of modules\r\n",
      "  --help [CORE_HELP]    Show most common flags and all task-specific flags\r\n",
      "  --help-all [CORE_HELP_ALL]\r\n",
      "                        Show all command line flags\r\n",
      "  --ShowFrequencies-log-file SHOWFREQUENCIES_LOG_FILE\r\n",
      "                        a custom log file; default: <task.default_log_file>\r\n",
      "  --log-file LOG_FILE   a custom log file; default: <task.default_log_file>\r\n",
      "  --ShowFrequencies-print-deps SHOWFREQUENCIES_PRINT_DEPS\r\n",
      "                        print task dependencies but do not run any task; this\r\n",
      "                        CSV parameter accepts a single integer value which\r\n",
      "                        sets the task recursion depth (0 means non-recursive)\r\n",
      "  --print-deps PRINT_DEPS\r\n",
      "                        print task dependencies but do not run any task; this\r\n",
      "                        CSV parameter accepts a single integer value which\r\n",
      "                        sets the task recursion depth (0 means non-recursive)\r\n",
      "  --ShowFrequencies-print-status SHOWFREQUENCIES_PRINT_STATUS\r\n",
      "                        print the task status but do not run any task; this\r\n",
      "                        CSV parameter accepts up to three values: 1. the task\r\n",
      "                        recursion depth (0 means non-recursive), 2. the depth\r\n",
      "                        of the status text of target collections (default: 0),\r\n",
      "                        3. a flag that is passed to the status text creation\r\n",
      "                        (default: '')\r\n",
      "  --print-status PRINT_STATUS\r\n",
      "                        print the task status but do not run any task; this\r\n",
      "                        CSV parameter accepts up to three values: 1. the task\r\n",
      "                        recursion depth (0 means non-recursive), 2. the depth\r\n",
      "                        of the status text of target collections (default: 0),\r\n",
      "                        3. a flag that is passed to the status text creation\r\n",
      "                        (default: '')\r\n",
      "  --ShowFrequencies-print-output SHOWFREQUENCIES_PRINT_OUTPUT\r\n",
      "                        print a flat list of output targets but do not run any\r\n",
      "                        task; this CSV parameter accepts up to two values: 1.\r\n",
      "                        the task recursion depth (0 means non-recursive), 2. a\r\n",
      "                        boolean flag that decides whether paths of file\r\n",
      "                        targets should contain file system schemes (default:\r\n",
      "                        True)\r\n",
      "  --print-output PRINT_OUTPUT\r\n",
      "                        print a flat list of output targets but do not run any\r\n",
      "                        task; this CSV parameter accepts up to two values: 1.\r\n",
      "                        the task recursion depth (0 means non-recursive), 2. a\r\n",
      "                        boolean flag that decides whether paths of file\r\n",
      "                        targets should contain file system schemes (default:\r\n",
      "                        True)\r\n",
      "  --ShowFrequencies-remove-output SHOWFREQUENCIES_REMOVE_OUTPUT\r\n",
      "                        remove task outputs but do not run any task by\r\n",
      "                        default; this CSV parameter accepts up to three\r\n",
      "                        values: 1. the task recursion depth (0 means non-\r\n",
      "                        recursive), 2. one of the modes 'i' (interactive), 'a'\r\n",
      "                        (all), 'd' (dry run) (default: 'i'), 3. a boolean flag\r\n",
      "                        that decides whether the task is run after outputs\r\n",
      "                        were removed (default: False)\r\n",
      "  --remove-output REMOVE_OUTPUT\r\n",
      "                        remove task outputs but do not run any task by\r\n",
      "                        default; this CSV parameter accepts up to three\r\n",
      "                        values: 1. the task recursion depth (0 means non-\r\n",
      "                        recursive), 2. one of the modes 'i' (interactive), 'a'\r\n",
      "                        (all), 'd' (dry run) (default: 'i'), 3. a boolean flag\r\n",
      "                        that decides whether the task is run after outputs\r\n",
      "                        were removed (default: False)\r\n",
      "  --ShowFrequencies-fetch-output SHOWFREQUENCIES_FETCH_OUTPUT\r\n",
      "                        copy all task outputs into a local directory but do\r\n",
      "                        not run any task; this CSV parameter accepts up to\r\n",
      "                        four values: 1. the task recursion depth (0 means non-\r\n",
      "                        recursive), 2. one of the modes 'i' (interactive), 'a'\r\n",
      "                        (all), 'd' (dry run) (default: 'i'), 3. the target\r\n",
      "                        directory (default: '.'), 4. a boolean flag that\r\n",
      "                        decides whether external outputs and outputs of\r\n",
      "                        external tasks should be fetched (default: False)\r\n",
      "  --fetch-output FETCH_OUTPUT\r\n",
      "                        copy all task outputs into a local directory but do\r\n",
      "                        not run any task; this CSV parameter accepts up to\r\n",
      "                        four values: 1. the task recursion depth (0 means non-\r\n",
      "                        recursive), 2. one of the modes 'i' (interactive), 'a'\r\n",
      "                        (all), 'd' (dry run) (default: 'i'), 3. the target\r\n",
      "                        directory (default: '.'), 4. a boolean flag that\r\n",
      "                        decides whether external outputs and outputs of\r\n",
      "                        external tasks should be fetched (default: False)\r\n",
      "  --ShowFrequencies-slow [SHOWFREQUENCIES_SLOW]\r\n",
      "                        before running, wait between 5 and 15 seconds;\r\n",
      "                        default: False\r\n",
      "  --slow [SLOW]         before running, wait between 5 and 15 seconds;\r\n",
      "                        default: False\r\n"
     ]
    }
   ],
   "source": [
    "%law run ShowFrequencies --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Run the ShowFrequencies task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we run the task and all its dependencies with a single command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mShowFrequencies_False_9ba313c494\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mMergeCounts_False_9ba313c494\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_6_False_d07face2c7\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_6_False_d07face2c7\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_5_False_675cf0b527\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_5_False_675cf0b527\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_4_False_c828a5a8e7\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_4_False_c828a5a8e7\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_3_False_21e31eaf1d\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_3_False_21e31eaf1d\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_2_False_5579299431\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_2_False_5579299431\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_1_False_50b924af96\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_1_False_50b924af96\u001b[0m   has status   \u001b[0;49;36mPENDING\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m\u001b[1;49;32mDone scheduling tasks\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mRunning Worker with 1 processes\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_6_False_d07face2c7\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_6_False_d07face2c7\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_5_False_675cf0b527\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_5_False_675cf0b527\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_4_False_c828a5a8e7\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_4_False_c828a5a8e7\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_3_False_21e31eaf1d\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_3_False_21e31eaf1d\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_2_False_5579299431\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_2_False_5579299431\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mFetchLoremIpsum\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mFetchLoremIpsum_1_False_50b924af96\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mCountChars\u001b[0m\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mCountChars_1_False_50b924af96\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mMergeCounts\u001b[0m\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mMergeCounts\u001b[0m\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mMergeCounts_False_9ba313c494\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;36mrunning\u001b[0m   \u001b[1;49;39m\u001b[0;49;32mShowFrequencies\u001b[0m\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "e: \u001b[0;49;32mxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\u001b[0m 9.6 %\n",
      "i: \u001b[0;49;32mxxxxxxxxxxxxxxxxxxxxxxxxxx\u001b[0m 8.5 %\n",
      "a: \u001b[0;49;32mxxxxxxxxxxxxxxxxxxxxxxxxx\u001b[0m 8.3 %\n",
      "o: \u001b[0;49;32mxxxxxxxxxxxxxxxxxxxxxxxx\u001b[0m 8.1 %\n",
      "t: \u001b[0;49;32mxxxxxxxxxxxxxxxxxxxxxxx\u001b[0m 7.8 %\n",
      "u: \u001b[0;49;32mxxxxxxxxxxxxxxxxxxxxxx\u001b[0m 7.4 %\n",
      "s: \u001b[0;49;33mxxxxxxxxxxxxxxxxxxxxx\u001b[0m 6.9 %\n",
      "n: \u001b[0;49;33mxxxxxxxxxxxxxxxxxx\u001b[0m 5.9 %\n",
      "r: \u001b[0;49;33mxxxxxxxxxxxxxxxxx\u001b[0m 5.6 %\n",
      "l: \u001b[0;49;33mxxxxxxxxxxxxxxx\u001b[0m 4.9 %\n",
      "m: \u001b[0;49;33mxxxxxxxxxxxx\u001b[0m 4.1 %\n",
      "d: \u001b[0;49;31mxxxxxxxxxxx\u001b[0m 3.8 %\n",
      "c: \u001b[0;49;31mxxxxxxxxxxx\u001b[0m 3.6 %\n",
      "p: \u001b[0;49;31mxxxxxxxxxx\u001b[0m 3.3 %\n",
      "g: \u001b[0;49;31mxxxxxxxx\u001b[0m 2.6 %\n",
      "h: \u001b[0;49;31mxxxx\u001b[0m 1.2 %\n",
      "b: \u001b[0;49;31mxxxx\u001b[0m 1.2 %\n",
      "w: \u001b[0;49;31mxxx\u001b[0m 1.1 %\n",
      "k: \u001b[0;49;31mxxx\u001b[0m 1.1 %\n",
      "f: \u001b[0;49;31mxxx\u001b[0m 1.0 %\n",
      "v: \u001b[0;49;31mxxx\u001b[0m 1.0 %\n",
      "y: \u001b[0;49;31mxxx\u001b[0m 0.9 %\n",
      "q: \u001b[0;49;31mxx\u001b[0m 0.8 %\n",
      "x: \u001b[0;49;31mxx\u001b[0m 0.6 %\n",
      "z: \u001b[0;49;31mx\u001b[0m 0.4 %\n",
      "j: \u001b[0;49;31mx\u001b[0m 0.1 %\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m[pid 27097] Worker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) \u001b[0;49;32mdone\u001b[0m      \u001b[1;49;39m\u001b[0;49;32mShowFrequencies\u001b[0m\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mInformed scheduler that task   \u001b[1;49;39mShowFrequencies_False_9ba313c494\u001b[0m   has status   \u001b[0;49;32mDONE\u001b[0m\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39mWorker Worker(salt=905796400, workers=1, host=the_host, username=marcel, pid=27097) was stopped. Shutting down Keep-Alive thread\u001b[0m\n",
      "\u001b[0;49;32mINFO\u001b[0m: \u001b[0;49;39mluigi-interface\u001b[0m - \u001b[0;49;39m\n",
      "===== Luigi Execution Summary =====\n",
      "\n",
      "Scheduled 14 tasks of which:\n",
      "* 14 ran successfully:\n",
      "    - 6 CountChars(...)\n",
      "    - 6 FetchLoremIpsum(...)\n",
      "    - 1 MergeCounts(...)\n",
      "    - 1 ShowFrequencies(...)\n",
      "\n",
      "This progress looks :) because there were no failed tasks or missing dependencies\n",
      "\n",
      "===== Luigi Execution Summary =====\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "%law run ShowFrequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The task execution should be successful within a few seconds. You can scroll through the output and read the logs to get a sense of the way luigi is building up the dependency tree, followed by the scheduling of tasks, and eventually closing with an execution summary.\n",
    "\n",
    "Also, you might want to add the ``--slow`` parameter to make the tasks somewhat slower in order to see the progress logs appearing in the output. This is of course not a feature of law, but only implemented by the tasks in this example 😉."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Check the status again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As above, we add `--print-status -1` again, to see the task status, represented by the existence of their output targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print task status with max_depth -1 and target_depth 0\r\n",
      "\r\n",
      "0 > \u001b[0;49;32mShowFrequencies\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "│\r\n",
      "└──1 > \u001b[0;49;32mMergeCounts\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_merged.json)\r\n",
      "   │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_1.json)\r\n",
      "   │  │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_1.txt)\r\n",
      "   │             \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_2.json)\r\n",
      "   │  │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_2.txt)\r\n",
      "   │             \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_3.json)\r\n",
      "   │  │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_3.txt)\r\n",
      "   │             \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_4.json)\r\n",
      "   │  │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_4.txt)\r\n",
      "   │             \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │  │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_5.json)\r\n",
      "   │  │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │  │\r\n",
      "   │  └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │           \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_5.txt)\r\n",
      "   │             \u001b[1;49;32mexistent\u001b[0m\r\n",
      "   │\r\n",
      "   └──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "      │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_6.json)\r\n",
      "      │       \u001b[1;49;32mexistent\u001b[0m\r\n",
      "      │\r\n",
      "      └──3 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "               \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_6.txt)\r\n",
      "                 \u001b[1;49;32mexistent\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "%law run ShowFrequencies --print-status -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `ShowFrequencies` task itself has no outputs. It is run **once**, but every time it is invoked, independent of the presence of a persistent file. The other tasks do have outputs, which we are going to delete in the next step."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Remove outputs interactively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, another interactive parameter to pass to `law run` commands is `--remove-output`. The passed value is interpreted as the recursion depth of dependent tasks whose output should be removed as well.\n",
    "\n",
    "However, in order to avoid removing files by mistake, law interactively asks for confirmation before irreversibly removing anything. The prompt looks like this:\n",
    "\n",
    "```shell\n",
    "> law run ShowFrequencies --remove-output N\n",
    "\n",
    "remove task output with max_depth N\n",
    "removal mode? [i*(interactive), d(dry), a(all)]\n",
    "```\n",
    "\n",
    "The default mode (marked with \\*) is *interactive* (type 'i'), which means that law traverses the task tree interactively and asks for confirmation on every target. *dry* mode (type 'd') traverses the tree without actually removing anything. The *all* mode should be handled with care. Once you type 'a', the outputs of all tasks down to the requested recursion depth are removed.\n",
    "\n",
    "To avoid interactive prompts in this example notebook, you can either do (though **not** recommended)\n",
    "\n",
    "```shell\n",
    "> echo a | law run ShowFrequencies --remove-output N\n",
    "```\n",
    "\n",
    "or add the mode with a comma to the value of the `--remove-output` parameter. Here, we only want to remove the outputs down to the `CountChars` task, i.e., at a depth of 2 (see the task tree above in the `--print-status` outputs). This way, the `FetchLoremIpsum` outputs are preserved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remove task output with max_depth 2\r\n",
      "selected \u001b[1;49;34mall mode\u001b[0m mode\r\n",
      "\r\n",
      "0 > \u001b[0;49;32mShowFrequencies\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "│\r\n",
      "└──1 > \u001b[0;49;32mMergeCounts\u001b[0m(\u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_merged.json)\r\n",
      "   │       \u001b[1;49;31mremoved\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │        \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_1.json)\r\n",
      "   │          \u001b[1;49;31mremoved\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=2, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │        \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_2.json)\r\n",
      "   │          \u001b[1;49;31mremoved\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=3, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │        \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_3.json)\r\n",
      "   │          \u001b[1;49;31mremoved\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=4, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │        \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_4.json)\r\n",
      "   │          \u001b[1;49;31mremoved\u001b[0m\r\n",
      "   │\r\n",
      "   ├──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=5, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "   │        \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_5.json)\r\n",
      "   │          \u001b[1;49;31mremoved\u001b[0m\r\n",
      "   │\r\n",
      "   └──2 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=6, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "            \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_6.json)\r\n",
      "              \u001b[1;49;31mremoved\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "%law run ShowFrequencies --remove-output 2,a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify your action by printing the status of, let's say, the first `CountChars` task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print task status with max_depth -1 and target_depth 0\r\n",
      "\r\n",
      "0 > \u001b[0;49;32mCountChars\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "│     \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/chars_1.json)\r\n",
      "│       \u001b[1;49;31mabsent\u001b[0m\r\n",
      "│\r\n",
      "└──1 > \u001b[0;49;32mFetchLoremIpsum\u001b[0m(\u001b[1;49;34mfile_index\u001b[0m=1, \u001b[1;49;34mslow\u001b[0m=False)\r\n",
      "         \u001b[0;49;36mLocalFileTarget\u001b[0m(\u001b[1;49;34mfs\u001b[0m=local_fs, \u001b[1;49;34mpath\u001b[0m=$DATA_PATH/loremipsum_1.txt)\r\n",
      "           \u001b[1;49;32mexistent\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "%law run CountChars --file-index 1 --print-status -1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}