docs/source/notebooks/upload-my-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Upload My Analysis to Flywheel\n",
    "**Date**:  26-Sept-2022  \n",
    "**Description**:  \n",
    "\n",
    "- This notebook provides a walk through to upload analyses generated on a local filesystem to Flywheel. Some neuroimaging analyses and workflows are not yet supported as Flywheel gears. For these workflows, the current workaround is to download your Flywheel analyses (or workflow inputs) run the workflow on your local machine, then upload the restuls (workflow outputs) back to flywheel as a new analysis. \n",
    "- It should be possible to run this notebook in any jupyter-compatible thrid party-platforms such as [google collab](https://colab.research.google.com/) or [mybinder.org](https://mybinder.org/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "- University of Colorado at Boulder Research Computing (CURC) account\n",
    "- Access to University of Colorado Flywheel Instance\n",
    "\n",
    "The following workbook should be run on CURC Blanca Compute. If you are unsure you have the correct permission or access to these resources please contact INC Data and Analysis team: Amy Hegarty [amy.hegarty@colorado.edu] or Lena Sherbakov [lena.sherbakov@colorado.edu]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CURC Jupyterhub \n",
    "Before launching this jupyter notebook, users should launch this session using Open OnDemand.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>TIP:</b> Follow the instructions on INC Documentation to get started with <a href=\"https://inc-documentation.readthedocs.io/en/dev/pl_and_blanca_basics.html#high-performance-compute-portal\">Jupyter Notebooks</a>.\n",
    "</div> \n",
    "\n",
    "We will be working on a large scratch system mounted only on Blanca compute nodes in this tutorial. If you do not have access to this filesystem you should select a similar large capacity scratch enviornment for analysis.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='setup'>Setup</a>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>TIP:</b> Please use the \"flywheel\" kernel for this tutorial. If you do not see a \"flywheel\" kernel, contact INC Data and Analysis team to install this environment.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Welcome to Intermountain Neuroimaging Consortium!\n"
     ]
    }
   ],
   "source": [
    "print(\"Welcome to Intermountain Neuroimaging Consortium!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python standard package come first\n",
    "import logging\n",
    "import os, platform, sys\n",
    "from zipfile import ZipFile\n",
    "from datetime import datetime\n",
    "\n",
    "# Third party packages come second\n",
    "import flywheel\n",
    "\n",
    "# add software paths\n",
    "sys.path.append('/projects/ics/software/flywheel-python/bids-client/')\n",
    "sys.path.append('/projects/ics/software/flywheel-python/')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets intialize a logger to keep track of the progress of our job (e.g. useful to keep track of runtime). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate a logger\n",
    "logging.basicConfig(level=logging.INFO)\n",
    "log = logging.getLogger('root')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets check we are on the correct computing system. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "host = os.getenv('HOSTNAME', os.getenv('COMPUTERNAME', platform.node())).split('.')[0]\n",
    "\n",
    "if \"bnode\" not in host:\n",
    "    log.error(\"Tutorial should be run on CURC high performance compute nodes: blanca\")\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flywheel API Key and Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get you API_KEY by following the steps described in the Flywheel SDK doc [here](https://flywheel-io.gitlab.io/product/backend/sdk/branches/master/python/getting_started.html#api-key)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>DANGER:</b> \n",
    "    Do NOT share your API key with anyone for any reason - it is the same as sharing your password and may break human subject participant confidentiality. ALWAYS obscure credentials from your code, especially when sharing with others/commiting to a shared repository.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_KEY = getpass('Enter API_KEY here: ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate the Flywheel API client either using the API_KEY provided by the user input above or by reading it from the environment variable `FW_KEY`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fw = flywheel.Client(API_KEY if 'API_KEY' in locals() else os.environ.get('FW_KEY'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check which Flywheel instance you have been authenticated against with the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log.info('You are now logged in as %s to %s', fw.get_current_user()['email'], fw.get_config()['site']['api_url'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often you will have to define a few constants in your notebook which serve as the inputs. Such constant for instance is the API_KEY that was used to instantiate the Flywheel client. Other examples could be a PROJECT_ID or PROJECT_LABEL that will be used to identify a specific project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROJECT_LABEL = 'MyProject'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are all the custom helper functions we have developed for use in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_project_id(fw, project_label):\n",
    "    \"\"\"Return the first project ID matching project_label\n",
    "    \n",
    "    Args:\n",
    "       fw (flywheel.Client): A flywheel client\n",
    "       project_label (str):  A Project label\n",
    "       \n",
    "    Returns:\n",
    "       (str): Project ID or None if no project found\n",
    "    \"\"\"\n",
    "    project = fw.projects.find_first(f'label={project_label}')\n",
    "    if project:\n",
    "        return project.id\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload My Analysis\n",
    "First, lets point to a project in Flywheel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project_id = get_project_id(fw, PROJECT_LABEL)\n",
    "if project_id:\n",
    "    print(f'Project ID is: {project_id}.')\n",
    "else:\n",
    "    print(f'No Project with label {PROJECT_LABEL} found.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we need to zip the contents of our analysis then upload those zipped directories to a flywheel analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = fw.get_project(project_id)\n",
    "\n",
    "# create new project level analysis\n",
    "analysis = project.add_analysis(label='CONN Analysis: ' + datetime.now(\" %x %X\"))\n",
    "\n",
    "# zip outputs...\n",
    "os.system('zip -R conn_analysis conn_analysis/ conn_analysis.mat')\n",
    "\n",
    "# upload output zipped directories\n",
    " analysis.upload_output('conn_analysis.zip')\n",
    "\n",
    "if os.path.exists('conn_inputs.zip'):\n",
    "    os.system('zip -R conn_inputs conn_inputs/ ')\n",
    "    analysis.upload_output('conn_inputs.zip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you can open Flywheel GUI to check your analysis was sucessfully uploaded."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (myenv)",
   "language": "python",
   "name": "myenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}