WP7 multiple scenario workflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WP7 executing multiple scenario workflow through DARE exec-api\n",
    "\n",
    "##### Using DARE components:\n",
    "\n",
    "* Execution API: [`https://testbed.project-dare.eu/exec-api`](https://testbed.project-dare.eu/exec-api)\n",
    "\n",
    "    * Includes execution of d4p workflows and specfem. Also calls for uploading, downloading and listing data files.\n",
    "\n",
    "* dispel4py Registry API: [`https://testbed.project-dare.eu/d4p-registry`](https://testbed.project-dare.eu/d4p-registry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overview\n",
    "\n",
    "1. Register [multiple-scenario-workflow.py](https://gitlab.com/project-dare/dare-api/blob/master/examples/wp7/multiple-scenario-workflow.py) in dispel4py registry\n",
    "2. Create dispel4py input json from [multiple_scenario_usecase.py/function execute](https://gitlab.com/project-dare/WP7_IS-ENES_Climate4Impact/blob/master/multiple_scenario_usecase.py)\n",
    "3. Submit workflow for execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constants and Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constant hostnames of exec-api and d4p-registry api\n",
    "EXEC_API_HOSTNAME = 'https://testbed.project-dare.eu/exec-api'\n",
    "D4P_REGISTRY_HOSTNAME = 'https://testbed.project-dare.eu/d4p-registry'\n",
    "\n",
    "# D4P-registry credentials\n",
    "REG_USERNAME = 'root'\n",
    "REG_PASSWORD = 'root'\n",
    "\n",
    "# Imports\n",
    "import json, os\n",
    "import sys\n",
    "import requests\n",
    "\n",
    "# Get helper_functions from previous directory\n",
    "import helper_functions as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Register [multiple-scenario-workflow.py](https://gitlab.com/project-dare/dare-api/blob/master/examples/wp7/multiple-scenario-workflow.py) in dispel4py registry"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get dispel4py registry credentials by logging in using username and password (1/4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fcd23cf60cb428f43c087ead76a121ac9a5ed918 {'Authorization': 'Token fcd23cf60cb428f43c087ead76a121ac9a5ed918'}\n"
     ]
    }
   ],
   "source": [
    "auth_token = F.login(REG_USERNAME, REG_PASSWORD, D4P_REGISTRY_HOSTNAME)\n",
    "header = F.get_auth_header(auth_token)\n",
    "print(auth_token, header)\n",
    "\n",
    "# Write token and hostnames to json\n",
    "creds = {}\n",
    "creds['D4P_REGISTRY_HOSTNAME'] = D4P_REGISTRY_HOSTNAME\n",
    "creds['EXEC_API_HOSTNAME'] = EXEC_API_HOSTNAME\n",
    "creds['header'] = header\n",
    "creds['REG_USERNAME'] = REG_USERNAME\n",
    "creds['REG_PASSWORD'] = REG_PASSWORD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Register a workspace (2/4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added workspace: WP7_Workspace\n",
      "Workspace URL: http://testbed.project-dare.eu/workspaces/516/\n",
      "Workspace ID: 516\n"
     ]
    }
   ],
   "source": [
    "workspace_url, workspace_id = F.create_workspace(\"\", \"WP7_Workspace\", \"\", creds)\n",
    "workspace_id = int(workspace_id)\n",
    "print('Workspace URL: ' + workspace_url)\n",
    "print('Workspace ID: ' + str(workspace_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Register a ProcessingElementSignature (3/4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added Processing Element: generic_workflow\n",
      "PESig resource URL: http://testbed.project-dare.eu/pes/443/\n"
     ]
    }
   ],
   "source": [
    "pe_url = F.create_pe(desc=\"\", name=\"generic_workflow\", conn=[], pckg=\"wp7_package\",\n",
    "            workspace=workspace_url, clone=\"\", peimpls=[], creds=creds)\n",
    "print('PESig resource URL: ' + str(pe_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Register a ProcessingElementImplementation (Python Code) (4/4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n",
      "  InsecureRequestWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added Processing Element Implementation: generic_workflow\n",
      "PE Implemenation ID: 427\n"
     ]
    }
   ],
   "source": [
    "# Online code\n",
    "# req = requests.get('https://gitlab.com/project-dare/dare-api/raw/master/examples/wp7/multiple-scenario-workflow.py')\n",
    "\n",
    "# impl_id = F.create_peimpl(desc=\"\", code=str(req.text),\n",
    "#                               parent_sig=pe_url, pckg=\"wp7_package\",\n",
    "#                               name=\"multiple_scenario_usecase\", workspace=workspace_url,\n",
    "#                               clone=\"\", creds=creds)\n",
    "\n",
    "# Local code\n",
    "impl_id = F.create_peimpl(desc=\"\", code=open('generic_workflow.py').read(),\n",
    "                              parent_sig=pe_url, pckg=\"wp7_package\",\n",
    "                              name=\"generic_workflow\", workspace=workspace_url,\n",
    "                              clone=\"\", creds=creds)\n",
    "\n",
    "print('PE Implemenation ID: ' + str(impl_id))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Create dispel4py input json from [multiple_scenario_usecase.py/function execute](https://gitlab.com/project-dare/WP7_IS-ENES_Climate4Impact/blob/master/multiple_scenario_usecase.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "$ python multiple_scenario_usecase.py\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload input json "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Zip input files and upload"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'OK!'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.system('zip -r input_C4I.zip input_C4I.json')\n",
    "F.upload(token=F.auth(), path='wp7-input', local_path='input_C4I.zip', creds=creds)\n",
    "\n",
    "#os.system('zip -r pe_enes.zip pe_enes.py')\n",
    "#F.upload(token=F.auth(), path='wp7-input', local_path='pe_enes.zip', creds=creds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List user file directories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploaded files......\n",
      "\n",
      "\n",
      "API LOCAL path: /home/mpiuser/sfs/uploads/Th1s4sY0urT0k3Nn_wp7-input\n",
      "Execution path: /home/mpiuser/sfs/d4p/uploads/Th1s4sY0urT0k3Nn_wp7-input\n",
      "\n",
      "\n",
      "API LOCAL path: /home/mpiuser/sfs/uploads/Th1s4sY0urT0k3Nn_wp6-input\n",
      "Execution path: /home/mpiuser/sfs/d4p/uploads/Th1s4sY0urT0k3Nn_wp6-input\n",
      "\n",
      "\n",
      "API LOCAL path: /home/mpiuser/sfs/uploads/Th1s4sY0urT0k3Nn_d4p-input\n",
      "Execution path: /home/mpiuser/sfs/d4p/uploads/Th1s4sY0urT0k3Nn_d4p-input\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Files generated from runs......\n",
      "\n",
      "\n",
      "Api Local path: /home/mpiuser/sfs/runs/Th1s4sY0urT0k3Nn_10-16-19--15:03:31_API-46c1106b-a104-42d3-be65-107179bcc1ec_d4p-openmpi-8ac2b7-launcher-xr4v4\n",
      "Execution path: /home/mpiuser/sfs/d4p/runs/Th1s4sY0urT0k3Nn_10-16-19--15:03:31_API-46c1106b-a104-42d3-be65-107179bcc1ec_d4p-openmpi-8ac2b7-launcher-xr4v4\n",
      "\n",
      "\n",
      "Api Local path: /home/mpiuser/sfs/runs/Th1s4sY0urT0k3Nn_10-16-19--15:13:07_API-65ae1d3a-8fb2-4800-97bd-0c3ea03622a3_d4p-openmpi-8ac2b7-launcher-9w4cm\n",
      "Execution path: /home/mpiuser/sfs/d4p/runs/Th1s4sY0urT0k3Nn_10-16-19--15:13:07_API-65ae1d3a-8fb2-4800-97bd-0c3ea03622a3_d4p-openmpi-8ac2b7-launcher-9w4cm\n",
      "\n",
      "\n",
      "Api Local path: /WP6_EPOS/specfem3d/specfem3d_input/OUTPUT_FILES/Th1s4sY0urT0k3Nn_demo_data\n",
      "Execution path: /home/mpiuser/sfs/specfem/Th1s4sY0urT0k3Nn_demo_data\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "resp = F.myfiles(token=F.auth(), creds=creds)\n",
    "_json = json.loads(resp)\n",
    "_json['run'] = sorted(_json['run'], key=lambda k: k['exec_path'])\n",
    "F.files_pretty_print(_json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_LOCAL_PATH = '/home/mpiuser/sfs/uploads/Th1s4sY0urT0k3Nn_wp7-input'\n",
    "API_EXEC_PATH = '/home/mpiuser/sfs/d4p/uploads/Th1s4sY0urT0k3Nn_wp7-input'\n",
    "API_LOCAL_PATH_LOG = \"/home/mpiuser/sfs/runs/Th1s4sY0urT0k3Nn_10-16-19--15:13:07_API-65ae1d3a-8fb2-4800-97bd-0c3ea03622a3_d4p-openmpi-8ac2b7-launcher-9w4cm\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### List files for certain directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = F._list(path=API_LOCAL_PATH, creds=creds)\n",
    "F._list_pretty_print(json.loads(resp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = F._list(path=API_LOCAL_PATH_LOG, creds=creds)\n",
    "F._list_pretty_print(json.loads(resp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILE = \"input_C4I.json\"\n",
    "LOG_FILE = \"logs.txt\"\n",
    "LOCAL_PATH = \"logs.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  (Optional) Download files to local file system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Dowloading....'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.download(path=API_LOCAL_PATH_LOG + '/' + LOG_FILE, creds=creds, local_path=LOCAL_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (Optional) Share files using B2DROP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "F.send2drop(token=F.auth(), creds=creds, path=API_LOCAL_PATH + '/' + FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(API_LOCAL_PATH + '/' + FILE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Submit workflow for execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data in submit_d4p:  {'user': 'root', 'pwd': 'root', 'impl_id': 427, 'pckg': 'wp7_package', 'wrkspce_id': 516, 'name': 'generic_workflow', 'n_nodes': 8, 'access_token': 'Th1s4sY0urT0k3Nn', 'reqs': 'https://gist.githubusercontent.com/davidath/d26b4848590e7e9556b52f93fd60e974/raw/801f539aeb1d1204b1c5277e6856e04154b69e19/requirements.txt'}\n",
      "https://testbed.project-dare.eu/exec-api\n",
      "OK!\n"
     ]
    }
   ],
   "source": [
    "F.submit_d4p(impl_id=impl_id, pckg=\"wp7_package\", workspace_id=workspace_id, pe_name=\"generic_workflow\",\n",
    "           token=F.auth(), creds=creds, target='simple', n_nodes=8, no_processes=8, iterations=1,\n",
    "           reqs='https://gist.githubusercontent.com/davidath/d26b4848590e7e9556b52f93fd60e974/raw/801f539aeb1d1204b1c5277e6856e04154b69e19/requirements.txt',\n",
    "           inputfile=API_EXEC_PATH + '/' + FILE)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Monitor calls\n",
    "F.monitor(creds=creds) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Monitor container status (temporal, more abstract representation of user jobs will be provided)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running containers...\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "resp = F.my_pods(token=F.auth(), creds=creds)\n",
    "F.pod_pretty_print(json.loads(resp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleanup (Delete workspace)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deleted workspace WP7_Workspace\n"
     ]
    }
   ],
   "source": [
    "F.delete_workspace('WP7_Workspace', creds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}