Tutorials/SBPLAT/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quickstart for the Seven Bridges Platform\n",
    "### Overview\n",
    "To introduce you to the major features of the Seven Bridges Platform, this QuickStart walks you through the process of a [Whole Exome Sequencing Analysis](https://igor.sbgenomics.com/public/apps#workflow/sevenbridges/public-apps/whole-exome-sequencing-gatk-2-3-9-lite). This API tutorial mirrors the [tutorial for the visual interface](http://docs.sevenbridges.com/docs/quickstart).\n",
    "\n",
    "This tutorial makes use of the [sevenbridges-python bindings](http://sevenbridges-python.readthedocs.io/en/latest/installation/).\n",
    "\n",
    "### Prerequisites\n",
    " 1. You need an account on one of the Seven Bridges Platforms. \n",
    " 2. You need your **authentication token**, and you need to pass this credential to the API. See <a href=\"Setup_API_environment.ipynb\">**the tutorial on setting up the API environment**</a> for details.\n",
    " \n",
    "## Imports\n",
    "We import the _Api_ class from the official sevenbridges-python bindings below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "import sevenbridges as sbg\n",
    "from sevenbridges.errors import Conflict\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the object\n",
    "The `Api` object needs to know your **auth\\_token** and the correct path. Here we assume you are using the .sbgrc file in your home directory. For other options see <a href=\"Setup_API_environment.ipynb\">Setup_API_environment.ipynb</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# [USER INPUT] specify platform {cgc, sbg}\n",
    "prof = 'sbpla'\n",
    "\n",
    "config_file = sbg.Config(profile=prof)\n",
    "api = sbg.Api(config=config_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a project\n",
    "_Projects_ are the foundation of any analysis on the the Platform. We can either work inside a project that has already been created or create a new project. Here we **create a new project**, but first **check that it doesn't already exist** to show both methods. The project name, billing group (we will use our free credits in the **Pilot Fund** billing group), and a project description will be sent in our API call. \n",
    "\n",
    "We start by listing all of your projects and your billing groups. Next, we create the JSON that will be passed to the API to create the project. The dictionary should include:\n",
    "* **billing_group** *Billing group* that will be charged for this project\n",
    "* **description**   (optional) Project description\n",
    "* **name**   Name of the project, may be *non-unique*<sup>1</sup>\n",
    "\n",
    "**After** creating the project, you can re-check the project list and get *additional* details assigned by the Platform, including (but not limited to):\n",
    "\n",
    "* **id**     _Unique_ identifier for the project, generated based on Project Name\n",
    "* **href**   Address<sup>2</sup> of the project.\n",
    "* **name**   Project name.\n",
    "\n",
    "<sup>1</sup> Please **don't** use non-unique *project names*. However, if you insist, the backend will allow it and assign a unique `id` to you project. This `id` is known as a [short name](http://docs.sevenbridges.com/docs/the-api#section-project-short-names)\n",
    "\n",
    "<sup>2</sup> This is the address where, by using API you can get this resource.\n",
    "\n",
    "#### PROTIPS\n",
    " * A detailed _recipe_ for creating projects is [here](../../Recipes/SBPLAT/projects_makeNew.ipynb)\n",
    " * Detailed documentation of this particular REST architectural style request is available [here](http://docs.sevenbridges.com/docs/create-a-new-project)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'api' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-369a7d2cfe1f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# check if this project already exists. LIST all projects and check for name match\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# Note that you can have more than one project with the same name. It is best practice to find things by ID.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m my_project_exists = [p for p in api.projects.query(limit=100).all() \n\u001b[0m\u001b[1;32m      7\u001b[0m               if p.name == new_project_name]      \n\u001b[1;32m      8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'api' is not defined"
     ]
    }
   ],
   "source": [
    "# [USER INPUT] Set project name here:\n",
    "new_project_name = 'Azzurri'                          \n",
    "      \n",
    "# check if this project already exists. LIST all projects and check for name match\n",
    "# Note that you can have more than one project with the same name. It is best practice to find things by ID.\n",
    "my_project_exists = [p for p in api.projects.query(limit=100).all() \n",
    "              if p.name == new_project_name]      \n",
    "              \n",
    "if my_project_exists:    # exploit fact that empty list is False\n",
    "    # If a project with the same name already exists, reuse the existing one\n",
    "    my_project = my_project_exists[0]\n",
    "\n",
    "    print('Project {} will be reused for next steps.'.format(my_project.id))\n",
    "    if hasattr(my_project, 'description'): # need to check if description has been entered\n",
    "        print('Project description: {} \\n'.format(my_project.description)) \n",
    "    \n",
    "else: \n",
    "    # What are my funding sources?\n",
    "    billing_groups = api.billing_groups.query()  \n",
    "\n",
    "    # Pick the first group (arbitrary)\n",
    "    print((billing_groups[0].name +\n",
    "           ' will be charged for computation and storage (if applicable) for your new project'))\n",
    "\n",
    "    # Set up the information for your new project\n",
    "    new_project = {\n",
    "            'billing_group': billing_groups[0].id,\n",
    "            'description': \"\"\"A project created by the API recipe (projects_makeNew.ipynb).\n",
    "                          This also supports **markdown**\n",
    "                          _Pretty cool_, right?\n",
    "                       \"\"\",\n",
    "            'name': new_project_name\n",
    "    }\n",
    "    \n",
    "    # CREATE the new project\n",
    "    my_project = api.projects.create(\n",
    "        name=new_project['name'], \n",
    "        billing_group=new_project['billing_group'],\n",
    "        description=new_project['description'],\n",
    "    )\n",
    "\n",
    "    print('Your new project {} has been created.'.format(my_project.name))\n",
    "    if hasattr(my_project, 'description'): # need to check if description has been entered\n",
    "        print('Project description: {} \\n'.format(my_project.description)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy input files from the _Public Reference Files_ repository\n",
    "[Public Reference Files](http://docs.sevenbridges.com/docs/file-repositories) is a repository of files maintained by the Seven Bridges Platform. It contains the latest and most frequently used reference genomes and annotation files, so you won't have to upload your own reference files every time you run a task. You can access this repository via the API as you would a project.\n",
    "\n",
    "Below, we will first list all our projects, then we'll list the files within the Public Reference Files repository<sup>3</sup>, and copy a file from Public Reference Files to your target project. We've hard-coded a list of file names to copy based on the tutorial.\n",
    "\n",
    "The critical information for this POST is the **file_id**. Note, you are allow to copy the same file as many times as you like. However, duplicates will be automatically have a prefix attached of (\\_1\\_, \\_2\\_, etc) depending on how many times you copy the file.\n",
    "\n",
    "#### PROTIPS \n",
    " * A detailed _recipe_ for copying Public Files is [here](../../Recipes/SBPLAT/files_copyFromPublicReference.ipynb)\n",
    " * Detailed documentation of this particular REST architectural style request is available [here](http://docs.sevenbridges.com/docs/copy-a-file)\n",
    "\n",
    "<sup>3</sup> Remember, files are only accessible **within** a project - here the Public Reference Files project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# [USER INPUT] Set files to copy here:  \n",
    "files_list = ['example_human_Illumina.pe_1.fastq',\n",
    "              'example_human_Illumina.pe_2.fastq']\n",
    "\n",
    "# Public reference and test files on the platform are available in the same project 'admin/sbg-public-data'\n",
    "source_project_id = 'admin/sbg-public-data'  \n",
    "\n",
    "# LIST all file names in your project and file objects from source project\n",
    "# Note that listing files in a project does not list subfolders\n",
    "my_file_names = [f.name for f in \n",
    "    api.files.query(limit=100, project=my_project).all()]\n",
    "\n",
    "source_files = api.files.query(\n",
    "    limit=100, project=source_project_id, names=files_list\n",
    ")\n",
    "    \n",
    "# Copy files if they don't already exist in my_project\n",
    "for f in source_files:\n",
    "    if f.name in my_file_names:\n",
    "        print('File {} already exists in second project, skipping'.format(f.name))\n",
    "    else:\n",
    "        print('File {} does not exist in {}; copying now'.format(\n",
    "              f.name, my_project.name))\n",
    "\n",
    "        new_file = f.copy(project=my_project, name=f.name)\n",
    "\n",
    "        # re-list files in target project to verify the copy worked\n",
    "        my_files = [f.name for f in api.files.query(\n",
    "                limit=100,project=my_project).all()]\n",
    "\n",
    "        if f.name in my_files:\n",
    "            print('Sucessfully copied one file!')\n",
    "        else:\n",
    "            print('Something went wrong...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is the meaning of this?\n",
    "Files are great, but without **metadata** they can be hard to manage. So here were are going to add metadata to these files. We will add one field that is _needed for the task_ and one to show _generality_.\n",
    "\n",
    "We've already listed all your files in the last cell. Here we will check the metadata for each one. A **detail**-call for files returns the following *attributes*:\n",
    "* **created_on** File creation date\n",
    "* **id**     _Unique_ identifier for the file\n",
    "* **name**   Name of the file, note this **is** metadata and can be _changed_\n",
    "* **href**   Address<sup>4</sup> of the file.\n",
    "* **modified_on** File modification date\n",
    "* **tags** File tags\n",
    "* **metadata** Dictionary of metadata\n",
    "* **origin**  Will link back to a *task* if this is an output file \n",
    "* **project** ID of the project the file is in\n",
    "* **parent** ID of the folder the file is in\n",
    "* **type** file or folder\n",
    "* **size** File size in bytes\n",
    "* **storage.type** Indicates whether the file is on platform built-in storage or on a connected cloud storage (Volume)\n",
    "* **storage.hosted_on_locations** Shows which cloud regions the file is stored in\n",
    "\n",
    "The **metadata** dictionary is both _changeable_ and _expandable_, but initially rather sparse with:\n",
    "* sample_id\n",
    "* platform\n",
    "* paired_end\n",
    "* library_id\n",
    "\n",
    "<sup>4</sup> This is the address where, by using API, you can get this resource.\n",
    "\n",
    "#### PROTIP\n",
    " * A detailed _recipe_ for detailing files is [here](../../Recipes/SBPLAT/files_detailOne.ipynb).\n",
    " * Detailed documentation of this particular REST architectural style request is available [here](http://docs.sevenbridges.com/docs/get-file-details)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "my_files = api.files.query(project=my_project, limit=100, names=files_list)\n",
    "\n",
    "for single_file in my_files.all():\n",
    "    print('You have selected file {} (size {} [bytes]). \\n'.format(\n",
    "        single_file.name, single_file.size\n",
    "    ))\n",
    "    print('The metadata in this file was: \\n %s \\n' % (single_file.metadata))\n",
    "    md = {\n",
    "        'platform_unit_id': '1',\n",
    "        'hasFlair':'True'\n",
    "    }\n",
    "\n",
    "    for k in md.keys():\n",
    "        single_file.metadata[k] = md[k]\n",
    "    \n",
    "    single_file.save()\n",
    "    \n",
    "    print('After the update, file metadata is: \\n %s \\n' % (single_file.metadata))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy reference files from the _Public Reference_\n",
    "Equivalent to the operation in **Copy input files from the _Public Reference_**, we are just looking for other file names. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# Files to copy\n",
    "ref_files = ['GRCh38.GRAF.Genome_Intervals.v1.bed',\n",
    "             'GRCh38.GRAF.Linear_Reference.v1.fa',\n",
    "             'GRCh38.GRAF.Linear_Reference.v1.fa.fai',\n",
    "             'GRCh38.GRAF.Pan_Genome_Reference.v1.vcf.gz',\n",
    "             'GRCh38.GRAF.Pan_Genome_Reference.v1.vcf.gz.tbi']\n",
    "\n",
    "\n",
    "# LIST all file names in your project and file objects from source project\n",
    "# Note that listing files in a project does not list subfolders\n",
    "my_file_names = [f.name for f in \n",
    "    api.files.query(limit=100, project=my_project).all()]\n",
    "\n",
    "source_files = api.files.query(\n",
    "    limit=100, project=source_project_id, names=ref_files\n",
    ")\n",
    "\n",
    "# Copy files if they don't already exist in my_project\n",
    "for f in source_files:\n",
    "    if f.name in my_file_names:\n",
    "        print('File {} already exists in second project, skipping'.format(f.name))\n",
    "    else:\n",
    "        print('File {} does not exist in {}; copying now'.format(\n",
    "              f.name, my_project.name))\n",
    "\n",
    "        new_file = f.copy(project=my_project, name=f.name)\n",
    "\n",
    "        # re-list files in target project to verify the copy worked\n",
    "        my_files = [f.name for f in api.files.query(\n",
    "                limit=100,project=my_project).all()]\n",
    "\n",
    "        if f.name in my_files:\n",
    "            print('Sucessfully copied one file!')\n",
    "        else:\n",
    "            print('Something went wrong...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Copy a Public App\n",
    "Seven Bridges maintains a [repository of publicly available apps](http://docs.sevenbridges.com/docs/public-apps) suitable for many different types of data analysis. These public apps, including tools and workflows, can be accessed via the API as part of the Public Reference project. They are also accessibly by a *visibility* property which can be set to `public`. Below, we use the first method to find apps within Public Reference project\n",
    "\n",
    "First, we'll list all our projects. Then, we'll list the apps within the Public Reference project. Lastly, we'll copy an app from the Public Reference project to the my\\_project. Here we also explicitly set _'limit':100_ inside the _query_. This helps speed up the auto-pagination feature within the object constructor.\n",
    "\n",
    "The critical information for this `POST` request is the **app_id**. Note, you are **not** allowed to copy the same app **and** assign the same name<sup>6</sup> more than once. If you change the name, it is ok. \n",
    "\n",
    "To make these results very obvious, use an empty project as your my\\_project (e.g. your <a href=projects_makeNew.ipynb> cookbook example project</a>) or add the _name_ arguement to something ridiculous like 'Orange Mocha Frapachino Maker'. In this example, we will handle these situations with a predefined error `Conflict` for this situation.\n",
    "\n",
    "<sup>6</sup> Note that setting the **name** of a copied app defines the app **id**, which must be unique on the platform. Saving over an existing ID will raise a `Conflict` error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# [USER INPUT] Set app name here\n",
    "# Note that you can have multiple apps or projects with the same name. It is best practice to reference entities by ID.\n",
    "a_name = 'GRAF Germline Variant Detection Workflow'\n",
    "\n",
    "for app in api.apps.query(visibility='public', limit=100).all():\n",
    "    if app.name == a_name:\n",
    "        public_app = app\n",
    "        break\n",
    "\n",
    "try:\n",
    "    new_app = public_app.copy(project=my_project)\n",
    "    print('App {} copied to Project {}.'.format(public_app.name, my_project.name))\n",
    "except Conflict:\n",
    "    new_app = [a for a in api.apps.query(project=my_project) if a.name == a_name][0]\n",
    "    print('App already exists in the destination project, reusing existing app.')\n",
    "        \n",
    "# re-list apps in target project to verify the copy worked\n",
    "my_app_names = [a.name for a in api.apps.query(project=my_project.id, limit=100).all()]\n",
    "\n",
    "if a_name in my_app_names:\n",
    "    print('Sucessfully copied or reused one app!')\n",
    "else:\n",
    "    print('Something went wrong...')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check app inputs\n",
    "\n",
    "First we can check what inputs are expected in our app. The app property `raw` contains a `dict` that has all the app information ready for grabs. Going through the `raw['inputs]'` can show us expected input IDs and types.\n",
    "Hints for input types: \n",
    "- The `[]` suffix, e.g. `File[]`, is a syntax denoting a list of items, in this case Files. \n",
    "- The `?` suffix, e.g. `File?`, is a syntax denoting an optional input, in this case an optional File. Another way to represent an optional type is, e.g. `['null', 'int']` for optional integer input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Expected inputs for app {}:'.format(new_app.name))\n",
    "for inp in new_app.raw['inputs']:\n",
    "    print('id: {}{}type: {}'.format(inp['id'].lstrip('#'), ' ' * (30 - len(inp['id'])), inp['type']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build & Start tasks\n",
    "Here, we use the copied reference files and set inputs. Note that input files are passed as a _file_ (or a _list_ of _files_). Here, there are no string, number or boolean inputs. For Apps that require these inputs, they would be the values, such as:\n",
    "\n",
    "```python\n",
    "inputs = {\n",
    "    'num_repititions' : 8,\n",
    "    'mask' : False, \n",
    "    'file_name' : 'output_backup.txt'\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "task_name = 'task created with quickstart.ipynb'\n",
    "\n",
    "# get the file objects and set them as file inputs\n",
    "# this is how a single file is set\n",
    "in_intervals = api.files.query(project=my_project, limit=100,\n",
    "                       names = [ref_files[0]])[0]\n",
    "in_linear_reference = api.files.query(project=my_project, limit=100,\n",
    "                       names=[ref_files[1]])[0]\n",
    "in_graph_reference = api.files.query(project=my_project, limit=100,\n",
    "                       names=[ref_files[3]])[0]\n",
    "\n",
    "# and here an array of files is set\n",
    "in_reads = api.files.query(project=my_project, limit=100,\n",
    "                       names=files_list)\n",
    "\n",
    "inputs = {\n",
    "    'in_intervals': in_intervals,\n",
    "    'in_linear_reference': in_linear_reference,\n",
    "    'in_graph_reference': in_graph_reference,\n",
    "    'in_reads': in_reads\n",
    "}\n",
    "\n",
    "my_task = api.tasks.create(name=task_name, project=my_project,\n",
    "                           app=new_app, inputs=inputs)\n",
    "\n",
    "# Check for errors and warnings\n",
    "if my_task.errors:\n",
    "    print(my_task.errors)\n",
    "# elif my_task.warnings:        # feature is in staging\n",
    "#     print(my_task.warnings)\n",
    "else:\n",
    "    print('Your task is good to go, launching!')\n",
    "    \n",
    "    # Start the task\n",
    "    my_task.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Print task status\n",
    "Here we poll the recently created task. With my_task.reload() we get latest information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "my_task.reload()\n",
    "print('Your task is in %s status' % (my_task.status))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for task completion\n",
    "Simple loop to ping for task completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# [USER INPUT] Set loop time (seconds):\n",
    "loop_time = 30\n",
    "flag = {'taskRunning': True}\n",
    "\n",
    "print('Pinging SBPLAT for task completion.')\n",
    "while flag['taskRunning']:\n",
    "    my_task.reload()\n",
    "    if my_task.status == 'COMPLETED':\n",
    "        flag['taskRunning'] = False\n",
    "        print('Task has completed, life is beautiful')\n",
    "    elif my_task.status  == 'FAILED':  \n",
    "        print('Task failed, can not continue')\n",
    "        raise KeyboardInterrupt\n",
    "    else:\n",
    "        print('Task status: {}'.format(my_task.status))\n",
    "        time.sleep(loop_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get task outputs\n",
    "Here we reload the task and print task outputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "my_task.reload()\n",
    "print(my_task.outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That’s it! We've executed a data analysis and obtained some results. We encourage you to try this procedure for yourself before getting started on your own data analyses. You can also visit our [API documentation](http://docs.sevenbridges.com/v1.0/page/api) to learn more about the Seven Bridges Platform and bringing your own tools."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}