Dammit_State_of_the_Software.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "nbpresent": {
     "id": "93f3e12c-39f7-41a0-9430-d4ed0437a8f4"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'height': 1080,\n",
       " 'scroll': True,\n",
       " 'start_slideshow_at': 'selected',\n",
       " 'theme': 'beige',\n",
       " 'transition': 'zoom',\n",
       " 'width': 1800}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from traitlets.config.manager import BaseJSONConfigManager\n",
    "path = \"/Users/camille/.jupyter/nbconfig/\"\n",
    "cm = BaseJSONConfigManager(config_dir=path)\n",
    "cm.update('livereveal', {\n",
    "              'theme': 'beige',\n",
    "              'transition': 'zoom',\n",
    "              'start_slideshow_at': 'selected',\n",
    "              'scroll': True,\n",
    "              'width': 1800,\n",
    "              'height': 1080,\n",
    "              \n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "fb59781b-4a22-4910-a27d-01307e4a1041"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# dammit: status and future directions\n",
    "## (aka dammit: state of the fart) \n",
    "### (aka \"dammit, I should release dammit 1.0”)\n",
    "\n",
    "--------------------------------------\n",
    "\n",
    "#### Camille Scott\n",
    "\n",
    "#### June 14, 2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Ze Meeting\n",
    "\n",
    "* Business to discuss?\n",
    "* dammit primer\n",
    "* What's new in 1.0?\n",
    "* Future directions\n",
    "* Discussion on annotation, community needs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "6f7f0b67-77a2-4def-acad-6278ef1de419"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# dammit primer\n",
    "\n",
    "<img align=\"left\" src=\"doc/_static/Character_Building.png\" width=\"300\" style=\"margin:15px\">\n",
    "\n",
    "dammit is a simple de novo transcriptome annotator. It was born out of the\n",
    "observations that annotation is mundane and annoying, all the individual pieces\n",
    "of the process exist already, and the existing solutions are overly complicated\n",
    "or rely on crappy non-free software.\n",
    "\n",
    "Science shouldn't suck for the sake of sucking, so dammit attempts\n",
    "to make this sucky part of the process suck a little less."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "cd451900-ad99-41b6-b4d5-6b8e383a8c61"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What should a good annotator even look like?\n",
    "\n",
    "Thoughts from 2015...\n",
    "\n",
    "* It should be easy to install and upgrade\n",
    "* It should only use Free software\n",
    "* It should make use of standard databases\n",
    "* It should output in reasonable formats\n",
    "* It should be relatively fast\n",
    "\n",
    "and of course,\n",
    "\n",
    "* it should try to be correct!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "5eeba8b3-2116-4aee-a367-863d0a8b74db"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### The Obligatory Flowchart\n",
    "<center>\n",
    "![The Workflow](doc/_static/workflow.svg)\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## dammit 1.0\n",
    "\n",
    "- Work on a candidate 1.0 release started... a while ago\n",
    "- If I recall, it started as \"fix one command on macOS,\" and then ballooned to \"refactor the whole thing\" -- sigh --\n",
    "- Overall goals are:\n",
    "   * Make dammit more easily maintainable and testable\n",
    "   * More easily support user requests\n",
    "   * Fix some useability issues and bu\n",
    "   gs\n",
    "   * Make it faster (and switch to shmlast)\n",
    "   * Allow more flexible pipelines\n",
    "   * Make it publishable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## What's Changed?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### BUSCO\n",
    "\n",
    "- BUSCOs developers really like changing the command line interface o_o\n",
    "- BUT, it has gained some useful features, and is now on V3\n",
    "- dammit 1.0 now supports v2 and v3 (v1 suppport has been dropped)\n",
    "- ...as well as *all* the databases\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center> ![](doc/_static/busco_dammit_dbs.png)<center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "### TransDecoder\n",
    "\n",
    "- Supporting version 3+ only now (they changed the output format)\n",
    "- and it's been out for a while\n",
    "- and supporting all the versions of all the things is difficult\n",
    "- (side note: this is a big problem for all bioinformatics dev)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "### Parallel\n",
    "\n",
    "- After lots of headaches, (I think) dammit fully supports GNU-parallel\n",
    "- Including on clusters:\n",
    "\n",
    "```\n",
    "dammit annotate test.fa --sshloginfile <name_of_loginfile>\n",
    "```\n",
    "\n",
    "The gnu-parallel docs have more on the format for this file (and I have an example kicking around somewhere, I think in the docs).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "---\n",
    "### Multiple Pipelines\n",
    "\n",
    "There are now three pipeline options:\n",
    "\n",
    "- default: everything except `lastal` against uniref90\n",
    "- `--quick`: basic stats, BUSCO, TransDecoder, and user provided databases (drops HMMER, Infernal, OrthoDB)\n",
    "- `--full`: All the things.\n",
    "\n",
    "WIP: Plugin support. Basic outline is sketched. For now, it's at least easier to add new tasks.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### Database Sanity Checks\n",
    "\n",
    "Several rather vexing errors ended up being caused by database downloads and/or preparations being interrupted. Databases are now MD5-checked after download.\n",
    "\n",
    "Downsides: this makes database status check a bit slower (not too bad though)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### WIP: OrthoDB\n",
    "\n",
    "- OrthoDB v9 has changed substantially\n",
    "- I'd like to more intelligently handle taxonomy levels when selecting hits from OrthoDB (lots of redundancy currently)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### User-facing Miscellany\n",
    "\n",
    "- Status output is now fully-compliant github markdown\n",
    "- Cleaned up a number of off-by-one errors in coordinates (I'm sure there are more...)\n",
    "- Support for a custom `config.json` at the CLI with `--config-file`\n",
    "- TODO: Figure out how to report BUSCO\n",
    "- TODO (maybe): HTML summary page?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### shmlast!\n",
    "\n",
    "- crb-blast has been fully extracated (sorry Richard)\n",
    "- Replaced by shmlast, which uses LAST and is written in Python (and maintained by me :))\n",
    "- fun dev note: shmlast is also implemented with pydoit, which means dammit can just import its tasks rather than having to run it separately\n",
    "- shmlast has been published in JOSS: http://joss.theoj.org/papers/3cde54de7dfbcada7c0fc04f569b36c7\n",
    "- Future directions: play around with better models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### Some Developer Miscellany\n",
    "\n",
    "- Tests are currently being switched from nose to pytest (because less overhead, particularly mental overhead, for me)\n",
    "- As mentioned before, codebase has undergone a yuge refactor\n",
    "    * Removed the `dammit dependencies` subcommand\n",
    "    * Instead, tasks with third-party software reqs subclass `DependentTask`, which exposes an interface to check for existence of command line tools\n",
    "    * ie, package dep checks with tasks: makes them more portable, and make dep checks happen on task creation rather than in a separate command\n",
    "    * Also makes testing easier\n",
    "- Docstrings almost everywhere\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Future of Dammit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "![](https://usedkey.files.wordpress.com/2013/11/witches-dancing.png)\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "### Features\n",
    "\n",
    "- Should dammit make more intelligent choices when filtering annotations?\n",
    "    * Figuring out a most likely annotation?\n",
    "     [ give filtered and full ]\n",
    "     [ jupyter recipes ]\n",
    "    * Graph-foo?\n",
    "- An HTML summary page?\n",
    "- What sort of output do users (biologists) really want? (Apparently: gene names)\n",
    "    * Is GFF3 useful, or a burden?\n",
    "    [ gene name -> transcript map, excel *shudder* importable ]\n",
    "    [ ortholog -> transcript map ]\n",
    "    [ functional annotations. check Meg Staton's ANGUS tutorial? ]\n",
    "    * Would plain CSV be better?\n",
    "- Contamination check with sourmash?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "\n",
    "[discussion]\n",
    "\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Keeping The Dream Alive\n",
    "\n",
    "- How should it be maintained?\n",
    "    * I can commit to periodic maintenance at least through the end of my PhD\n",
    "    * But what then?\n",
    "- How to improve community involvement\n",
    "    * Better onboarding\n",
    "    * Better / any dev docs\n",
    "    * Better contributor guidelines\n",
    "    * The Big Refactor helps here -- code is easier to grok\n",
    "    * [ Give it its own org on github ]\n",
    "    * [ Titus thoughts: point Tim at it. Look at osf-cli for ideas. Sand rough corners and allow it to propogate through poweruser vectors (ie Lisa, Harriet).\n",
    "    \n",
    "- How to make third-party reqs more future-proof?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "\n",
    "[discussion]\n",
    "\n",
    "</center>"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:py3.dammit]",
   "language": "python",
   "name": "conda-env-py3.dammit-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}