docs/Tutorials/Quickstart/cli_quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cromwell-tools Command Line Interface Quickstart\n",
    "\n",
    "This notebook will help you get familiar with the Cromwell-tools' CLI (command line interface). After walking through \n",
    "this notebook, you should be able to work with Cromwell engine using the `cromwell-tools` command in your terminal.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you successfully installed the `cromwell-tools`, you shall be bale to verify if the CLI is added to your available commands by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check all of the available commands and help text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could check all of the available commands by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: cromwell-tools [-h] [-V]\n",
      "                      {submit,wait,status,abort,release_hold,query,health} ...\n",
      "\n",
      "positional arguments:\n",
      "  {submit,wait,status,abort,release_hold,query,health}\n",
      "                        sub-command help\n",
      "    submit              submit help\n",
      "    wait                wait help\n",
      "    status              status help\n",
      "    abort               abort help\n",
      "    release_hold        release_hold help\n",
      "    query               query help\n",
      "    health              health help\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  -V, --version         show program's version number and exit\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could also check the detailed help message, for instance, to check the help message of `submit`, you could:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: cromwell-tools submit [-h] [--url URL] [--username USERNAME]\n",
      "                             [--password PASSWORD]\n",
      "                             [--secrets-file SECRETS_FILE]\n",
      "                             [--service-account-key SERVICE_ACCOUNT_KEY] -w\n",
      "                             WDL_FILE -i INPUTS_FILES [INPUTS_FILES ...]\n",
      "                             [-d DEPENDENCIES [DEPENDENCIES ...]]\n",
      "                             [-o OPTIONS_FILE] [-l LABEL_FILE]\n",
      "                             [-c COLLECTION_NAME] [--on-hold ON_HOLD]\n",
      "                             [--validate-labels VALIDATE_LABELS]\n",
      "\n",
      "Submit a WDL workflow on Cromwell.\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  --url URL             The URL to the Cromwell server. e.g.\n",
      "                        \"https://cromwell.server.org/\"\n",
      "  --username USERNAME   Cromwell username for HTTPBasicAuth.\n",
      "  --password PASSWORD   Cromwell password for HTTPBasicAuth.\n",
      "  --secrets-file SECRETS_FILE\n",
      "                        Path to the JSON file containing username, password,\n",
      "                        and url fields.\n",
      "  --service-account-key SERVICE_ACCOUNT_KEY\n",
      "                        Path to the JSON key file for authenticating with\n",
      "                        CaaS.\n",
      "  -w WDL_FILE, --wdl-file WDL_FILE\n",
      "                        Path to the workflow source file to submit for\n",
      "                        execution.\n",
      "  -i INPUTS_FILES [INPUTS_FILES ...], --inputs-files INPUTS_FILES [INPUTS_FILES ...]\n",
      "                        Path(s) to the input file(s) containing input data in\n",
      "                        JSON format, separated by space.\n",
      "  -d DEPENDENCIES [DEPENDENCIES ...], --deps-file DEPENDENCIES [DEPENDENCIES ...]\n",
      "                        Path to the Zip file containing dependencies, or a\n",
      "                        list of raw dependency files to be zipped together\n",
      "                        separated by space.\n",
      "  -o OPTIONS_FILE, --options-file OPTIONS_FILE\n",
      "                        Path to the Cromwell configs JSON file.\n",
      "  -l LABEL_FILE, --label-file LABEL_FILE\n",
      "                        Path to the JSON file containing a collection of\n",
      "                        key/value pairs for workflow labels.\n",
      "  -c COLLECTION_NAME, --collection-name COLLECTION_NAME\n",
      "                        Collection in SAM that the workflow should belong to,\n",
      "                        if use CaaS.\n",
      "  --on-hold ON_HOLD     Whether to submit the workflow in \"On Hold\" status.\n",
      "  --validate-labels VALIDATE_LABELS\n",
      "                        Whether to validate cromwell labels.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard Authentication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cromwell-tools supports 4 types of authentication when talking to Cromwell:\n",
    "\n",
    "- Authenticate with Cromwell using HTTPBasicAuth (username and password)\n",
    "- Authenticate with Cromwell using HTTPBasicAuth (secret JSON file)\n",
    "- Authenticate with Cromwell using OAuth (service account JSON key file)\n",
    "- Authenticate with Cromwell with no Auth\n",
    "\n",
    "As you might have noticed, most of the commands that cromwell-tools provides share a same set of auth-related arguments:\n",
    "\n",
    "- `--url`\n",
    "- `--username`\n",
    "- `--password`\n",
    "- `--secrets-file`\n",
    "- `--service-account-key`\n",
    "\n",
    "You would like to choose the right auth combinations depending on the Cromwell engine you work with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate with Cromwell using HTTPBasicAuth (username and password)\n",
    "\n",
    "Passing only the `--username` and `--password` besides the `--url` indicates that you want to authenticate with a HTTPBasicAuth-protected Cromwell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"d3dfa1a0-1134-46dc-9b29-335e645f3be9\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.xxx.broadinstitute.org\" \\\n",
    "--username \"xxx\" \\\n",
    "--password \"xxx\" \\\n",
    "--wdl \"Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"Examples/inputs.json\" \\\n",
    "--deps-file \"Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate with Cromwell using HTTPBasicAuth (secret JSON file)\n",
    "\n",
    "Passing the username and password every time you run the command sounds like a burden. You could save the efforts by storing the HTTPBasic Authentication credentials as well as the cromwell URL into a JSON file (e.g. `secrets.json`) following the format:\n",
    "\n",
    "```JSON\n",
    "{\n",
    "    \"url\": \"url\",\n",
    "    \"username\": \"username\",\n",
    "    \"password\": \"password\"\n",
    "}\n",
    "```\n",
    "\n",
    "Now you can pass the secret file path to the cromwell-tools:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"25dc480d-6755-4b6b-b932-ded2ff634754\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--secrets-file \"secrets.json\" \\\n",
    "--wdl \"Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"Examples/inputs.json\" \\\n",
    "--deps-file \"Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate with Cromwell using OAuth (Google Cloud service account JSON key file)\n",
    "\n",
    "You could also use cromwell-tools to talk to a Crowmell that is using OAuth2. You will need to pass in the Service Account JSON key file to the `--service-account-key` argument. This file is usually generated by Google Cloud and downloaded from the Cloud console, which should have sufficient permission to talk to te OAuth-enbaled Cromwell. \n",
    "\n",
    "Specifically, if you are working with Cromwell-as-a-Service (i.e. CaaS), you will need to specify `--collection-name` when submitting a workflow, which indicates a valid collection in [SAM](https://github.com/broadinstitute/sam) that the workflow should belong to.\n",
    "\n",
    "Here is an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"a42e54ba-84e4-4305-be90-feeae33b7fc4\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.caas-prod.broadinstitute.org\" \\\n",
    "--service-account-key \"/path/to/your/service-account-json-key.json\" \\\n",
    "--collection-name \"your-collection\" \\\n",
    "--wdl \"Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"Examples/inputs.json\" \\\n",
    "--deps-file \"Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authenticate with Cromwell with no Auth\n",
    "\n",
    "Sometimes you have to work with a Cromwell that does not have any authentication layer in front of it, no worries, just skip the auth arguments and fill the `--url`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"d3dfa1a0-1134-46dc-9b29-335e645f3be9\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.xxx.broadinstitute.org\" \\\n",
    "--wdl \"Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"Examples/inputs.json\" \\\n",
    "--deps-file \"Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explain the arguments and the `submit` command\n",
    "\n",
    "### Using short optional arguments\n",
    "You may have noticed that some of the arguments of `submit` have shorter versions, and yes, they are identical to their long counterparts: the following 2 examples are identical!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"a42e54ba-84e4-4305-be90-feeae33b7fc4\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.caas-prod.broadinstitute.org\" \\\n",
    "--service-account-key \"/path/to/your/service-account-json-key.json\" \\\n",
    "--collection-name \"your-collection\" \\\n",
    "--wdl \"Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"Examples/inputs.json\" \\\n",
    "--deps-file \"Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"a42e54ba-84e4-4305-be90-feeae33b7fc4\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.caas-prod.broadinstitute.org\" \\\n",
    "--service-account-key \"/path/to/your/service-account-json-key.json\" \\\n",
    "-c \"your-collection\" \\\n",
    "-w \"Examples/hello_world.wdl\" \\\n",
    "-i \"Examples/inputs.json\" \\\n",
    "-d \"Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workflow labels and options\n",
    "\n",
    "Workflow labels and options files are optional, you can pass the JSON files for your workflow when submitting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"a42e54ba-84e4-4305-be90-feeae33b7fc4\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.caas-prod.broadinstitute.org\" \\\n",
    "--service-account-key \"/path/to/your/service-account-json-key.json\" \\\n",
    "--collection-name \"your-collection\" \\\n",
    "--wdl \"Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"Examples/inputs.json\" \\\n",
    "--deps-file \"Examples/helloworld.wdl\" \\\n",
    "--label-file \"Examples/options.json\" \\\n",
    "--options-file \"Examples/labels.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using URLs as the file paths\n",
    "\n",
    "So far we have been passing the local (absolute or relative) paths as arguments, but we could also use HTTP/HTTPS paths, crowmell-tools will download and compose them for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\":\"13f7577a-c496-4770-8f3e-0f085f4edac0\",\"status\":\"Submitted\"}\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools submit \\\n",
    "--url \"https://cromwell.caas-prod.broadinstitute.org\" \\\n",
    "--service-account-key \"/path/to/your/service-account-json-key.json\" \\\n",
    "--collection-name \"your-collection\" \\\n",
    "--wdl \"https://raw.githubusercontent.com/broadinstitute/cromwell-tools/v2.0.0/notebooks/Quickstart/Examples/hello_world.wdl\" \\\n",
    "--inputs-files \"https://raw.githubusercontent.com/broadinstitute/cromwell-tools/v2.0.0/notebooks/Quickstart/Examples/inputs.json\" \\\n",
    "--deps-file \"https://raw.githubusercontent.com/broadinstitute/cromwell-tools/v2.0.0/notebooks/Quickstart/Examples/helloworld.wdl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The `wait` command\n",
    "\n",
    "The `wait` command is a special command that helps polling and monitoring the workflow(s) in the Cromwell engine. You can use it to keep track of a list of workflows(they are NOT necessarily to be submitted by cromwell-tools!). By default `wait` will print verbose information while polling, but you can configure its behavior using the following arguemnts and flag(s):\n",
    "\n",
    "- `--timeout-minutes`\n",
    "- `--poll-interval-seconds`\n",
    "- `--silent`\n",
    "\n",
    "**Note: failure of one of the workflows will cause the whole polling process to be terminated, so please only poll one workflow if you are not sure if the workflow can succeed.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- polling from cromwell ---\n",
      "Workflow a42e54ba-84e4-4305-be90-feeae33b7fc4 returned status Succeeded\n",
      "Workflow 13f7577a-c496-4770-8f3e-0f085f4edac0 returned status Succeeded\n",
      "All workflows succeeded!\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cromwell-tools wait \\\n",
    "--url \"https://cromwell.caas-prod.broadinstitute.org\" \\\n",
    "--service-account-key \"/path/to/your/service-account-json-key.json\" \\\n",
    "--poll-interval-seconds 10 \\\n",
    "a42e54ba-84e4-4305-be90-feeae33b7fc4 \\\n",
    "13f7577a-c496-4770-8f3e-0f085f4edac0 \\\n",
    "d9d8dc18-d462-46f6-a39d-b68b20dfb5ab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other commands\n",
    "\n",
    "You can check the available arugments of other commands by using `cromwell-tools COMMAND -h`, other commands are just simple mappings of the corresponding Cromwell endpoints. Note that the `query` command is not yet implemented in the CLI, please use the Python API client instead!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}