docs/source/getting_started/running_at_scale.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running Algorithms at Scale\n",
    "\n",
    "In order to run algorithms in the scaled-up cloud compute environment, they must first be \"registered\" in the Algorithm Catalog. This will make them available to other MAAP users, clearly define their inputs and outputs, and prepare them to be run easily in the Data Processing System (DPS).\n",
    "\n",
    "![Running Algorithms overview in context diagram](_static/running_algorithms_overview.png)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register an Algorithm\n",
    "\n",
    "### Clone a test algorithm\n",
    "\n",
    "This is an example algorithm you can use for this getting started guide:\n",
    "https://github.com/MAAP-Project/dps-unit-test\n",
    "\n",
    "![Demo GitHub repo](_static/example_alg1.png)\n",
    "In the repo there are a few files that you will typically have, or which are required:\n",
    "\n",
    " - `algorithm_config.yml` is a required file that has a description of the inputs and outputs of the algorithm along with other parameters like the run command.\n",
    " - `run_test.sh` is the run command for this algorithm. It is typical to have a shell script to tell the system how to run the algorithm and set some environmental variables.\n",
    "\n",
    "1. Make a new folder for your test algorithm. Open a terminal here (File > New > Terminal or use the blue \"+\" button above the Jupyter file browser).\n",
    "![New Folder](_static/clone_demo1.png)\n",
    "\n",
    "2. Copy the Github clone link from https://github.com/MAAP-Project/dps-unit-test\n",
    "![Copy .git link](_static/clone_demo2.png)\n",
    "\n",
    "3. Open the built-in Jupyter Github UI to the left of the file browser. Choose \"Clone a Repository\" and paste in the .git link you copied from the Github repository.\n",
    "![Clone a Repository](_static/clone_demo3.png)\n",
    "![Paste .git link](_static/clone_demo4.png)\n",
    "\n",
    "4. You should see a new folder created with the repo you cloned. If you browse to that folder and open up the Jupyter Github UI again, it will show you some info about that repo.\n",
    "![Algorithm folder was created](_static/clone_demo5.png)\n",
    "![Browse to folder](_static/clone_demo6.png)\n",
    "![Look at Github UI](_static/clone_demo7.png)\n",
    "\n",
    "5. If you want to make changes to the code and have your own copy of it to register, Clone the code into your MAAP GitLab. The git link to your code is indicated in the `algorithm_config.yml`. If you would prefer to skip this for now, leave the repository_url in `algorithm_config.yml` pointed at the \"root\" user (`repository_url: https://repo.dit.maap-project.org/root/dps-unit-test.git`)\n",
    "\n",
    "6. Rename the algorithm to personalize it. You do this by opening up the `algorithm_config.yml` file and changing the `algo_name` field.\n",
    "![Rename the Algorithm](_static/rename_alg.png)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Register the algorithm\n",
    "\n",
    "1. Make sure code is ready and saved -> right click file -> “Register as MAS Algorithm”\n",
    "![Register as algorithm contextual menu](_static/register-as-alg.png)\n",
    "2. This automatically creates `algorithm_config.yaml` file with the presets if it is not already present (which, in this example case, it is present). There is only one for any directory. At this point you would normally edit the configuration file, then repeat step 1 and click \"OK\" to register. For this example we did this in step 5 in the previous section.\n",
    "3. Outputs (if any) should be written to a folder named `outputs`. There are none in the example we are using here.\n",
    "\n",
    ".. note:: \n",
    "    It can take some time to register an algorithm. You can determine if it has completed when you see it appear in the Jobs UI (see below) or in the menus under DPS/MAS Operations > List Algorithms."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the Algorithm as a Job and Monitor it\n",
    "\n",
    "### The Jobs UI\n",
    "MAAP is configured to run up to 4,000 concurrent jobs. There are two additional ways to run a job: via the Jobs UI in the Launcher, or via a call to the maap-py Python library.\n",
    "\n",
    "The Jobs UI will let you run and monitor jobs easily. You can find full documentation in the system reference guide for [the Jobs UI](../system_reference_guide/ade_custom_extensions/jobsui.ipynb).\n",
    "You can also find specific documentation on [how to submit jobs](../system_reference_guide/faq/job_submit.ipynb) and [how to monitor jobs](../system_reference_guide/faq/job_check_status.ipynb) in the System Reference Guide FAQs.\n", 
    "\n",
    "![job_ui_access](../_static/jobs_ui_access.png)\n",
    "\n",
    "Some alternative methods of running the job are found below.\n",
    "\n",
    "### Pop-up\n",
    "- Click DPS/MAS Operations menu -> Execute DPS Job\n",
    "- Select your algorithm from the dropdown\n",
    "- A new popup will ask for inputs; if it doesn’t take inputs, the popup will say so.\n",
    "- Click OK again to view the ID for the job just submitted.\n",
    "\n",
    "OR\n",
    "\n",
    "### maap-py\n",
    "Import the `maap-py` library: if in Jupyter, click the small blue MAAP button in the top left corner to automatically insert code. If using a script, add these lines manually at the top of your notebook: \n",
    "```\n",
    "from maap.maap import MAAP\n",
    "maap = MAAP()\n",
    "```\n",
    "\n",
    "Pass your algorithm’s name, version, required inputs, and username to the function `maap.submitJob (identifier is job- algo_name:algo_version)`\n",
    "Check result: `maap.getJobResult()`\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}