mlin's new demux launcher

[tomkinsc]

This is a branch off @mlin's branch proposed in #809. It adds an applet title so the launcher job includes more than just the run_id in its title. It also adds auto-scaling, currently commented out as the instance type passed to dx run seems to propagate to override instance types specified in individual tasks. The L# directories are now placed below reads/.

[dpark01]

Is this a non-deterministic Travis test failure? (pasting it here for future reference in case the error disappears on re-run).

_______ ERROR at teardown of test_kraken_multi[TestMetagenomicsViralMix] _______
[gw1] linux -- Python 3.6.5 /home/travis/build/broadinstitute/viral-ngs/tools/conda-tools/default/bin/python
def reset():
>       shutil.rmtree(new_tempdir)
conftest.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tools/conda-tools/default/lib/python3.6/shutil.py:480: in rmtree
    _rmtree_safe_fd(fd, path, onerror)
tools/conda-tools/default/lib/python3.6/shutil.py:438: in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
topfd = 11, path = '/tmp/pytest-of-travis/pytest-0/popen-gw1/test-function0'
onerror = <function rmtree.<locals>.onerror at 0x7f004cc5f1e0>
    def _rmtree_safe_fd(topfd, path, onerror):
        names = []
        try:
            names = os.listdir(topfd)
        except OSError as err:
            err.filename = path
            onerror(os.listdir, path, sys.exc_info())
        for name in names:
            fullname = os.path.join(path, name)
            try:
                orig_st = os.stat(name, dir_fd=topfd, follow_symlinks=False)
                mode = orig_st.st_mode
            except OSError:
                mode = 0
            if stat.S_ISDIR(mode):
                try:
                    dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
                except OSError:
                    onerror(os.open, fullname, sys.exc_info())
                else:
                    try:
                        if os.path.samestat(orig_st, os.fstat(dirfd)):
                            _rmtree_safe_fd(dirfd, fullname, onerror)
                            try:
                                os.rmdir(name, dir_fd=topfd)
                            except OSError:
                                onerror(os.rmdir, fullname, sys.exc_info())
                        else:
                            try:
                                # This can only happen if someone replaces
                                # a directory with a symlink after the call to
                                # stat.S_ISDIR above.
                                raise OSError("Cannot call rmtree on a symbolic "
                                              "link")
                            except OSError:
                                onerror(os.path.islink, fullname, sys.exc_info())
                    finally:
                        os.close(dirfd)
            else:
                try:
>                   os.unlink(name, dir_fd=topfd)
E                   FileNotFoundError: [Errno 2] No such file or directory: 'libgkl_compression2251932993111688424.so'
tools/conda-tools/default/lib/python3.6/shutil.py:436: FileNotFoundError

[dpark01]

I had noticed a while ago that I've had difficulty getting hdd2 AWS/DX instances to run. @mlin can you confirm if these are being phased out? Should we stick to ssd2 instances instead?

[dpark01]

Note for future reference... that Travis failure was indeed non-deterministic. I'm not exactly sure how "No such file or directory: 'libgkl_compression2251932993111688424.so'" could possibly be non-deterministic, but ... something to keep an eye on. I'll open an issue.

[dpark01]

Once any remaining technical items are sorted out and the whole system has been hand-tested on dx with a few different flowcell sizes (miseq, hs2500rr, nova), these changes look good from a review standpoint.