examples/load-project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using MLRun Projects and GIT\n",
    "  --------------------------------------------------------------------\n",
    "\n",
    "Loading full project with multiple functions and workflow and working wit Git.\n",
    "\n",
    "#### **notebook how-to's**\n",
    "* Load a project with multiple functions from Git\n",
    "* Run automated workflows (using KubeFlow)\n",
    "* Update, maintain and debug code "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='top'></a>\n",
    "#### **steps**\n",
    "**[Load project from Git or Archive](#load-project)**<br>\n",
    "**[Run a pipeline workflow](#run-pipeline)**<br>\n",
    "**[Updating the project and code](#update-project)**<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlrun import load_project, code_to_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='load-project'></a>\n",
    "## Load project from Git or Archive\n",
    "\n",
    "Projects can be stored in a Git repo or in a tar archive (on object store like S3, V3IO).\n",
    "\n",
    "`load_project(context, url)` will load/clone the project to the local context dir and build the project object from the `project.yaml` file in the git/archive root directory. \n",
    "\n",
    "> Note: If URL is not specified it will use the context and search for Git repo inside it, or use the `init_git=True` flag to initialize a Git repo in the target context directory.\n",
    "\n",
    "You can also clone the code into a dir using a CLI commands:\n",
    "\n",
    "`mlrun project my-proj/ -u git://github.com/mlrun/demo-xgb-project.git`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os import path\n",
    "from pathlib import Path\n",
    "\n",
    "# source Git Repo\n",
    "# YOU SHOULD fork this to your account and use the fork if you plan on modifying the code\n",
    "url = 'git://github.com/mlrun/demo-xgb-project.git' # refs/tags/v0.4.7'\n",
    "\n",
    "# alternatively can use tar files, e.g.\n",
    "#url = 'v3io:///users/admin/tars/src-project.tar.gz'\n",
    "\n",
    "# change if you want to clone into a different dir, can use clone=True to override the dir content\n",
    "project_dir = path.join(str(Path.home()), 'my_proj')\n",
    "proj = load_project(project_dir, url, clone=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><b> TL;DR You can just jump to [running the project](#run-cmd) now</b>\n",
    "\n",
    "## Play with the project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/User/my-proj\n"
     ]
    }
   ],
   "source": [
    "# if you are not in the project dir, change dir into the project dir\n",
    "%cd {project_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'git://github.com/mlrun/demo-xgb-project.git'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proj.source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the project object, note it contains lists of `functions` and `workflows` which will be used in the project. Functions can be local to the project or referenced to (via a URL to .ipynb, .py, .yaml file and/or container image). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: iris\n",
      "functions:\n",
      "- url: ./src/iris.yaml\n",
      "  name: xgb\n",
      "- url: https://raw.githubusercontent.com/mlrun/mlrun/master/examples/xgb_serving.ipynb\n",
      "  name: serving\n",
      "workflows:\n",
      "- name: main\n",
      "  path: src/workflow.py\n",
      "artifacts: []\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(proj.to_yaml())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-08 22:26:52,756 function spec saved to path: src/iris.yaml\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f0ac5dfe128>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can update the function .py and .yaml from a notebook (code + spec)\n",
    "# the \"code_output\" option will generate a .py file from our notebook which can be used for src control and local runs\n",
    "xgbfn = code_to_function('xgb', filename='notebooks/train-xgboost.ipynb' ,kind='job', code_output='src/iris.py')\n",
    "\n",
    "# tell the builder to clone this repo into the function container \n",
    "xgbfn.spec.build.source = './'\n",
    "xgbfn.export('src/iris.yaml')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "kind: job\n",
      "metadata:\n",
      "  name: xgb\n",
      "  tag: ''\n",
      "  hash: 6d79dd2f263f32a4d741f9362b247d00a47e50d9\n",
      "  project: iris\n",
      "  categories: []\n",
      "spec:\n",
      "  command: ''\n",
      "  args: []\n",
      "  image: ''\n",
      "  volumes: []\n",
      "  volume_mounts: []\n",
      "  env: []\n",
      "  default_handler: ''\n",
      "  entry_points:\n",
      "    iris_generator:\n",
      "      name: iris_generator\n",
      "      doc: ''\n",
      "      parameters:\n",
      "      - name: context\n",
      "      outputs: []\n",
      "      lineno: 16\n",
      "    xgb_train:\n",
      "      name: xgb_train\n",
      "      doc: ''\n",
      "      parameters:\n",
      "      - name: context\n",
      "      - name: dataset\n",
      "      - name: model_name\n",
      "        default: model.bst\n",
      "      - name: max_depth\n",
      "        default: 6\n",
      "      - name: num_class\n",
      "        default: 10\n",
      "      - name: eta\n",
      "        default: 0.2\n",
      "      - name: gamma\n",
      "        default: 0.1\n",
      "      - name: steps\n",
      "        default: 20\n",
      "      outputs: []\n",
      "      lineno: 25\n",
      "    plot_iter:\n",
      "      name: plot_iter\n",
      "      doc: ''\n",
      "      parameters:\n",
      "      - name: context\n",
      "      - name: iterations\n",
      "      - name: col\n",
      "        default: accuracy\n",
      "      - name: num_bins\n",
      "        default: 10\n",
      "      outputs: []\n",
      "      lineno: 62\n",
      "  description: ''\n",
      "  build:\n",
      "    functionSourceCode: IyBHZW5lcmF0ZWQgYnkgbnVjbGlvLmV4cG9ydC5OdWNsaW9FeHBvcnRlcgoKaW1wb3J0IHdhcm5pbmdzCndhcm5pbmdzLnNpbXBsZWZpbHRlcihhY3Rpb249J2lnbm9yZScsIGNhdGVnb3J5PUZ1dHVyZVdhcm5pbmcpCgppbXBvcnQgeGdib29zdCBhcyB4Z2IKaW1wb3J0IG9zCmZyb20gc2tsZWFybi5kYXRhc2V0cyBpbXBvcnQgbG9hZF9pcmlzCmZyb20gc2tsZWFybi5tb2RlbF9zZWxlY3Rpb24gaW1wb3J0IHRyYWluX3Rlc3Rfc3BsaXQKaW1wb3J0IG51bXB5IGFzIG5wCmZyb20gc2tsZWFybi5tZXRyaWNzIGltcG9ydCBhY2N1cmFjeV9zY29yZQpmcm9tIG1scnVuLmFydGlmYWN0cyBpbXBvcnQgVGFibGVBcnRpZmFjdCwgUGxvdEFydGlmYWN0CmltcG9ydCBwYW5kYXMgYXMgcGQKCgpkZWYgaXJpc19nZW5lcmF0b3IoY29udGV4dCk6CiAgICBpcmlzID0gbG9hZF9pcmlzKCkKICAgIGlyaXNfZGF0YXNldCA9IHBkLkRhdGFGcmFtZShkYXRhPWlyaXMuZGF0YSwgY29sdW1ucz1pcmlzLmZlYXR1cmVfbmFtZXMpCiAgICBpcmlzX2xhYmVscyA9IHBkLkRhdGFGcmFtZShkYXRhPWlyaXMudGFyZ2V0LCBjb2x1bW5zPVsnbGFiZWwnXSkKICAgIGlyaXNfZGF0YXNldCA9IHBkLmNvbmNhdChbaXJpc19kYXRhc2V0LCBpcmlzX2xhYmVsc10sIGF4aXM9MSkKICAgIGNvbnRleHQubG9nZ2VyLmluZm8oJ3NhdmluZyBpcmlzIGRhdGFmcmFtZSB0byB7fScuZm9ybWF0KGNvbnRleHQub3V0X3BhdGgpKQogICAgY29udGV4dC5sb2dfYXJ0aWZhY3QoVGFibGVBcnRpZmFjdCgnaXJpc19kYXRhc2V0JywgZGY9aXJpc19kYXRhc2V0KSkKICAgIAoKZGVmIHhnYl90cmFpbihjb250ZXh0LCAKICAgICAgICAgICAgICBkYXRhc2V0PScnLAogICAgICAgICAgICAgIG1vZGVsX25hbWU9J21vZGVsLmJzdCcsCiAgICAgICAgICAgICAgbWF4X2RlcHRoPTYsCiAgICAgICAgICAgICAgbnVtX2NsYXNzPTEwLAogICAgICAgICAgICAgIGV0YT0wLjIsCiAgICAgICAgICAgICAgZ2FtbWE9MC4xLAogICAgICAgICAgICAgIHN0ZXBzPTIwKToKCiAgICBkZiA9IHBkLnJlYWRfY3N2KGRhdGFzZXQpCiAgICBYID0gZGYuZHJvcChbJ2xhYmVsJ10sIGF4aXM9MSkKICAgIHkgPSBkZlsnbGFiZWwnXQogICAgCiAgICBYX3RyYWluLCBYX3Rlc3QsIFlfdHJhaW4sIFlfdGVzdCA9IHRyYWluX3Rlc3Rfc3BsaXQoWCwgeSwgdGVzdF9zaXplPTAuMikKICAgIGR0cmFpbiA9IHhnYi5ETWF0cml4KFhfdHJhaW4sIGxhYmVsPVlfdHJhaW4pCiAgICBkdGVzdCA9IHhnYi5ETWF0cml4KFhfdGVzdCwgbGFiZWw9WV90ZXN0KQoKICAgIHBhcmFtID0geyJtYXhfZGVwdGgiOiBtYXhfZGVwdGgsCiAgICAgICAgICAgICAiZXRhIjogZXRhLCAibnRocmVhZCI6IDQsCiAgICAgICAgICAgICAibnVtX2NsYXNzIjogbnVtX2NsYXNzLAogICAgICAgICAgICAgImdhbW1hIjogZ2FtbWEsCiAgICAgICAgICAgICAib2JqZWN0aXZlIjogIm11bHRpOnNvZnRwcm9iIn0KCiAgICB4Z2JfbW9kZWwgPSB4Z2IudHJhaW4ocGFyYW0sIGR0cmFpbiwgc3RlcHMpCgogICAgcHJlZHMgPSB4Z2JfbW9kZWwucHJlZGljdChkdGVzdCkKICAgIGJlc3RfcHJlZHMgPSBucC5hc2FycmF5KFtucC5hcmdtYXgobGluZSkgZm9yIGxpbmUgaW4gcHJlZHNdKQoKICAgIGNvbnRleHQubG9nX3Jlc3VsdCgnYWNjdXJhY3knLCBmbG9hdChhY2N1cmFjeV9zY29yZShZX3Rlc3QsIGJlc3RfcHJlZHMpKSkKICAgIGNvbnRleHQubG9nX2FydGlmYWN0KCdtb2RlbCcsIGJvZHk9Ynl0ZXMoeGdiX21vZGVsLnNhdmVfcmF3KCkpLCAKICAgICAgICAgICAgICAgICAgICAgICAgIGxvY2FsX3BhdGg9bW9kZWxfbmFtZSwgbGFiZWxzPXsnZnJhbWV3b3JrJzogJ3hnYm9vc3QnfSkKICAgIAogICAgCmltcG9ydCBtYXRwbG90bGliCmltcG9ydCBtYXRwbG90bGliLnB5cGxvdCBhcyBwbHQKZnJvbSBpbyBpbXBvcnQgQnl0ZXNJTwoKZGVmIHBsb3RfaXRlcihjb250ZXh0LCBpdGVyYXRpb25zLCBjb2w9J2FjY3VyYWN5JywgbnVtX2JpbnM9MTApOgogICAgZGYgPSBwZC5yZWFkX2NzdihCeXRlc0lPKGl0ZXJhdGlvbnMuZ2V0KCkpKQogICAgeCA9IGRmWydvdXRwdXQue30nLmZvcm1hdChjb2wpXQogICAgZmlnLCBheCA9IHBsdC5zdWJwbG90cyhmaWdzaXplPSg2LDYpKQogICAgbiwgYmlucywgcGF0Y2hlcyA9IGF4Lmhpc3QoeCwgbnVtX2JpbnMsIGRlbnNpdHk9MSkKICAgIGF4LnNldF94bGFiZWwoJ0FjY3VyYWNjeScpCiAgICBheC5zZXRfeWxhYmVsKCdDb3VudCcpCiAgICBjb250ZXh0LmxvZ19hcnRpZmFjdChQbG90QXJ0aWZhY3QoJ215ZmlnJywgYm9keT1maWcpKQoK\n",
      "    source: ./\n",
      "    base_image: mlrun/mlrun\n",
      "    commands:\n",
      "    - pip install sklearn\n",
      "    - pip install xgboost\n",
      "    - pip install matplotlib\n",
      "    code_origin: https://github.com/mlrun/demo-xgb-project.git#32ab2068eed70f995ad13a94e3f2da6733715f48\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# read specific function spec\n",
    "print(proj.func('xgb').to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run a project function locally "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-08 22:27:23,655 artifact path is not defined or is local, artifacts will not be visible in the UI\n",
      "[mlrun] 2020-06-08 22:27:23,661 starting run xgb-iris_generator uid=ec0ed9e3acf64e649a5142864a439d48  -> http://mlrun-api:8080\n",
      "[mlrun] 2020-06-08 22:27:23,689 starting local run: /tmp/tmp4ek9qp38.py # iris_generator\n",
      "[mlrun] 2020-06-08 22:27:24,574 .out_path will soon be deprecated, use .artifact_path\n",
      "[mlrun] 2020-06-08 22:27:24,574 saving iris dataframe to \n",
      "[mlrun] 2020-06-08 22:27:24,594 log artifact iris_dataset at iris_dataset.csv, size: 2776, db: Y\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style> \n",
       ".dictlist {\n",
       "  background-color: #b3edff; \n",
       "  text-align: center; \n",
       "  margin: 4px; \n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer; \n",
       "  background-color: #ffe6cc; \n",
       "  text-align: left; \n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #ffe6cc;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "  \n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "  \n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }  \n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "  \n",
       "  \n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>iris</td>\n",
       "      <td><div title=\"ec0ed9e3acf64e649a5142864a439d48\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/iris/jobs/ec0ed9e3acf64e649a5142864a439d48/info\" target=\"_blank\" >...4a439d48</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Jun 08 22:27:24</td>\n",
       "      <td>completed</td>\n",
       "      <td>xgb-iris_generator</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=admin</div><div class=\"dictlist\">kind=</div><div class=\"dictlist\">owner=admin</div><div class=\"dictlist\">host=jupyter-65887d7ffb-5jsn2</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultef9c5332\" title=\"/files/my-proj/iris_dataset.csv\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultef9c5332-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultef9c5332-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultef9c5332\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultef9c5332-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to track results use .show() or .logs() or in CLI: \n",
      "!mlrun get run ec0ed9e3acf64e649a5142864a439d48 --project iris , !mlrun logs ec0ed9e3acf64e649a5142864a439d48 --project iris\n",
      "[mlrun] 2020-06-08 22:27:24,645 run executed, status=completed\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<mlrun.model.RunObject at 0x7f0ac5dfe550>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from mlrun import run_local, new_task\n",
    "run_local(new_task(handler='iris_generator'), proj.func('xgb'), workdir='./')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='update-project'></a>\n",
    "## Updating the project and code\n",
    "\n",
    "A user can update the code using the standard Git process (commit, push, ..), if you update/edit the project object you need to run `proj.save()` which will update the `project.yaml` file in your context directory, followed by pushing your updates.\n",
    "\n",
    "You can use `proj.push(branch, commit_message, add=[])` which will do the work for you (save the yaml, commit updates, push)\n",
    "\n",
    "> Note: you must push updates before you build functions or run workflows since the builder will pull the code from the git repo.\n",
    "\n",
    "If you are using containerized Jupyter you may need to first set your Git parameters, e.g. using the following commands:\n",
    "\n",
    "```\n",
    "git config --global user.email \"<my@email.com>\"\n",
    "git config --global user.name \"<name>\"\n",
    "git config --global credential.helper store\n",
    "```\n",
    "\n",
    "After that you would need to login once to git with your password as well as restart the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RUN THIS ONLY IF YOU ARE IN YOUR FORK !!\n",
    "# proj.push('master', 'some edits')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to pull changes done by others use `proj.pull()`, if you need to update the project spec (since the yaml file changed) use `proj.reload()` and for updating the local/remote function specs use `proj.sync_functions()` (or add `sync=True` to `.reload()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proj.pull()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='run-pipeline'></a>\n",
    "## Run a pipeline workflow\n",
    "You can check the [workflow.py](src/workflow.py) file to see how functions objects are initialized and used (by name) inside the workflow.\n",
    "The `workflow.py` file has two parts, initialize the function objects and define pipeline dsl (connect the function inputs and outputs).\n",
    "\n",
    "> Note the pipeline can include CI steps like building container images and deploying models.\n",
    "\n",
    "### Initializing the functions (e.g. mount them on the v3io fabric)\n",
    "```python\n",
    "def init_functions(functions: dict, project=None, secrets=None):\n",
    "    for f in functions.values():\n",
    "        f.apply(mount_v3io())\n",
    "        \n",
    "```\n",
    "<br>\n",
    "\n",
    "### Workflow DSL:\n",
    "```python\n",
    "@dsl.pipeline(\n",
    "    name='My XGBoost training pipeline',\n",
    "    description='Shows how to use mlrun.'\n",
    ")\n",
    "def kfpipeline(\n",
    "        eta=[0.1, 0.2, 0.3], gamma=[0.1, 0.2, 0.3]\n",
    "):\n",
    "    # first step build the function container\n",
    "    builder = funcs['xgb'].deploy_step(with_mlrun=False)\n",
    "\n",
    "    # use xgb.iris_generator function to generate data (container image from the builder)\n",
    "    ingest = funcs['xgb'].as_step(name='ingest_iris', handler='iris_generator',\n",
    "        image=builder.outputs['image'],\n",
    "        outputs=['iris_dataset'])\n",
    "\n",
    "    # use xgb.xgb_train function to train on the data (from the generator step)\n",
    "    train = funcs['xgb'].as_step(name='xgb_train', handler='xgb_train',\n",
    "        image=builder.outputs['image'],\n",
    "        hyperparams={'eta': eta, 'gamma': gamma},\n",
    "        selector='max.accuracy',\n",
    "        inputs={'dataset': ingest.outputs['iris_dataset']},\n",
    "        outputs=['model'])\n",
    "\n",
    "    # deploy the trained model using a nuclio real-time function\n",
    "    deploy = funcs['serving'].deploy_step(models={'iris_v1': train.outputs['model']})\n",
    "```\n",
    "\n",
    "<a id='run-cmd'></a>\n",
    "### Run\n",
    "use the `run` method to execute a workflow, you can provide alternative arguments and specify the default target for workflow artifacts.<br>\n",
    "The workflow ID is returned and can be used to track the progress or you can use the hyperlinks\n",
    "\n",
    "> Note: The same command can be issued through CLI commands:<br>\n",
    "    `mlrun project my-proj/ -r main -p \"v3io:///users/admin/mlrun/kfp/{{workflow.uid}}/\"`\n",
    "\n",
    "The dirty flag allow us to run a project with uncommited changes (when the notebook is in the same git dir it will always be dirty)<br>\n",
    "The `watch` flag will wait until the pipeline completes and prints a summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-08 22:29:08,165 WARNING!, you seem to have uncommitted git changes, use .push()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/conda/lib/python3.6/site-packages/kfp/components/_data_passing.py:168: UserWarning: Missing type name was inferred as \"JsonArray\" based on the value \"[0.1, 0.2, 0.3]\".\n",
      "  warnings.warn('Missing type name was inferred as \"{}\" based on the value \"{}\".'.format(type_name, str(value)))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Experiment link <a href=\"https://dashboard.default-tenant.app.yh55.iguazio-cd2.com/pipelines/#/experiments/details/3e6e4f0d-553e-4d80-8eb2-a7fa520e62b0\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run link <a href=\"https://dashboard.default-tenant.app.yh55.iguazio-cd2.com/pipelines/#/runs/details/032e6d59-6bfe-4ee7-bcf6-1fb26e5db550\" target=\"_blank\" >here</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[mlrun] 2020-06-08 22:29:08,410 Pipeline run id=032e6d59-6bfe-4ee7-bcf6-1fb26e5db550, check UI or DB for progress\n",
      "[mlrun] 2020-06-08 22:29:08,411 waiting for pipeline run completion\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h2>Run Results</h2>Workflow 032e6d59-6bfe-4ee7-bcf6-1fb26e5db550 finished, status=Succeeded<br>click the hyper links below to see detailed results<br><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>uid</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td><div title=\"bf795aa8910c4d07851bd45bb6dcd8c0\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/iris/jobs/bf795aa8910c4d07851bd45bb6dcd8c0/info\" target=\"_blank\" >...b6dcd8c0</a></div></td>\n",
       "      <td>Jun 08 22:31:17</td>\n",
       "      <td>completed</td>\n",
       "      <td>xgb_train</td>\n",
       "      <td><div class=\"dictlist\">best_iteration=1</div><div class=\"dictlist\">accuracy=1.0</div></td>\n",
       "      <td><div title=\"v3io:///users/admin/mlrun/kfp/032e6d59-6bfe-4ee7-bcf6-1fb26e5db550/1/model.bst\">model</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"files/v3io/users/admin/mlrun/kfp/032e6d59-6bfe-4ee7-bcf6-1fb26e5db550/iteration_results.csv\">iteration_results</div></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td><div title=\"713af4e27a2b48f5ad31c9506d72ef7e\"><a href=\"https://mlrun-ui.default-tenant.app.yh55.iguazio-cd2.com/projects/iris/jobs/713af4e27a2b48f5ad31c9506d72ef7e/info\" target=\"_blank\" >...6d72ef7e</a></div></td>\n",
       "      <td>Jun 08 22:31:03</td>\n",
       "      <td>completed</td>\n",
       "      <td>ingest_iris</td>\n",
       "      <td></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"result\" title=\"files/v3io/users/admin/mlrun/kfp/032e6d59-6bfe-4ee7-bcf6-1fb26e5db550/iris_dataset.csv\">iris_dataset</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'032e6d59-6bfe-4ee7-bcf6-1fb26e5db550'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proj.run('main', arguments={}, artifact_path='v3io:///users/admin/mlrun/kfp/{{workflow.uid}}/', dirty=True, watch=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Replacing the source path to speed debug\n",
    "\n",
    "Instead of updating Git anytime we modify code we can build the code from the shared file system on the cluster (the build container will mount to the same location with the code instead of reading from Git).\n",
    "\n",
    "We need to change the project source to point to the shared file system URL of our context directory (e.g. v3io), and we can re-run the workflow. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "proj.source = 'v3io:///users/admin/my-proj'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[back to top](#top)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}