examples/tutorials/demo-vipsession.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Execute VIP Jobs from your Computer\n",
    "\n",
    "This Notebook uses the `VipSession` class from the [VIP Python client](https://github.com/virtual-imaging-platform/VIP-python-client/), which allows to run VIP applications on local datasets. In this tutorial you will learn how to:\n",
    "- **Create** a VipSession instance;\n",
    "- **Perform** an *upload-run-download* procedure (see image below) to get your outputs from VIP;\n",
    "- **Parallelize** your executions on VIP;\n",
    "- **Remove** your temporary data from VIP;\n",
    "- **Resume & repeat** previous VIP executions.\n",
    "\n",
    "For more advanced use of the Python client (*e.g.*, running different pipelines on the same dataset), please [read the doc on GitHub](https://github.com/virtual-imaging-platform/VIP-python-client/#vipsession).\n",
    "\n",
    "<img src=\"imgs/Upload_Run_Download.png\" alt=\"Procedure\" height=\"250\" title=\"Procedure for the Python Client\"/>\n",
    "\n",
    "\n",
    "*__N.B.__: In this Notebook, commands starting with \"`!`\" will only run on Linux distributions (including the Binder instance).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python libraries used in this tutorial\n",
    "from pathlib import *\n",
    "from vip_client import VipSession"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initiate your Session \n",
    "*The VIP API key can be genereated in your own account settings. This short procedure can be found in the [documentation](https://github.com/virtual-imaging-platform/VIP-python-client/#get-a-vip-api-key).* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Handshake with VIP\n",
    "VipSession.init(api_key=\"VIP_API_KEY\"); # Paste your VIP API key here"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*VIP jobs will be launched through a persistent `VipSession` instance. Providing a session name is a good practice.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a VipSession object with session name: \"Demo-VipSession\"\n",
    "my_session = VipSession(\"Demo-VipSession\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The *upload-run-download* procedure"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload your dataset on VIP"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Local folder `./data` contains the entire dataset*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_dir = Path(\"data\")\n",
    "! tree {input_dir}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The VipSession method `upload_inputs()` is meant to upload the full dataset on VIP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_session.upload_inputs(input_dir);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch the Application"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get the Pipeline Identifier"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Search for available applications on VIP "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VipSession.show_pipeline(\"cquest\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*If the previous cell displays \"`No pipeline found`\", please subscribe to the MR Spectroscopy group in your __account settings__ on the [VIP portal](https://vip.creatis.insa-lyon.fr/).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pipeline identifier = App/Version (/!\\ mind the case)\n",
    "pipeline_id = \"CQUEST/0.2-egi\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get your Input Settings"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the pipeline description to know which inputs are required by the application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VipSession.show_pipeline(pipeline_id)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide the input settings as a dictionnary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input values can take several formats:\n",
    "input_settings = { \n",
    "    \"zipped_folder\" : \"data/basis.zip\", # String\n",
    "    \"parameter_file\" : input_dir / \"parameters.txt\", # PathLib object\n",
    "    \"data_file\" : [file for file in (input_dir/\"signals\").iterdir()] # List\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*A **list** of values (`[...]`) submits __parallel__ jobs on VIP. In this example, all signals (`data_file`) will be processed in parallel with the same `parameter_file` and `zipped_folder`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the previous settings as strings to see the list of files in `data_file`\n",
    "import json\n",
    "print(\"input_settings =\",\n",
    "    json.dumps(indent=3, obj={\n",
    "        key: [str(v) for v in value] if isinstance(value, list) else str(value) for key, value in input_settings.items()\n",
    "        }\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch & monitor executions on VIP"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "VIP executions are launched with `launch_pipeline()`. Parallel *jobs* (*e.g.* processing the signals) are launched at once and grouped in a single *workflow*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_session.launch_pipeline(pipeline_id, input_settings);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*You can monitor the workflow progression on https://vip.creatis.insa-lyon.fr/* ...\n",
    "\n",
    "... or wait until all jobs are over on this terminal with `monitor_workflows()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_session.monitor_workflows(refresh_time=10);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download your Results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When all jobs are over, the outputs are downloaded at once with `download_outputs()`. Output files that have already been downloaded will be ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_session.download_outputs();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*If `output_dir` was not specified when instanciating `my_session`, the outputs are stored at default location (recommended for beginners)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree {my_session.output_dir}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comments\n",
    "1. This 4-step procedure (`upload_inputs()` -> `launch_pipeline()` -> `monitor_workflows()` -> `download_outputs()`) can be performed with a single command using `run_session()`:\n",
    "```python \n",
    "    my_session = VipSession(\n",
    "        session_name = \"Demo-VipSession\",\n",
    "        input_dir = input_dir,\n",
    "        pipeline_id = pipeline_id,\n",
    "        input_settings = input_settings\n",
    "    ).run_session(refresh_time=10);\n",
    "```\n",
    "2. Setting all properties (*e.g.*, `input_dir`, `pipeline_id`, `input_settings`) at instantiation (like above) allows early detection of common mistakes (*e.g.*, missing parameters or input files) before running VIP executions."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove temporary data from VIP"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*After the download, **your input and output data are still on VIP** (https://vip.creatis.insa-lyon.fr/)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Inputs are in:\", my_session.vip_input_dir, \"\\n\", \n",
    "      \"Outputs are in:\", my_session.vip_output_dir)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please remove the temporary data from VIP with `finish()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_session.finish();"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the session backups"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the backup file"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*After each step, session data are automatically backed up in its output directory* "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree {Path(my_session.output_dir).parent}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The JSON file `session_data.json` contains everything you need to repeat the same executions on VIP*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! head {Path(my_session.output_dir) / \"session_data.json\"} "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeat previous executions"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can restore a previous VipSession instance using its session name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_session = VipSession(\"Demo-VipSession\") # Name of the previous session"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `run_session()` shortcut to launch the full *Upload-Run-Download* procedure from this new VipSession instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_session.run_session(refresh_time=10);"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the output files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! tree {new_session.output_dir}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End this tutorial"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output data downloaded on your computer is yours to remove"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "! rm -r vip_outputs"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}