pipeline DB

[julienguy]

This PR is a major refactor of the pipeline.

• The pipeline now relies on a database to schedule the tasks to be run.
• The database can be sqlite or postgres , both working and tested at scale at NERSC.
• All tasks implemented and tested, from preprocessing to redshifts.

There is a nice command line

usage: desi_pipe <command> [options]

Where supported commands are:
   create   Create a new production.
   env      Print current production location.
   update   Update an existing production.
   getready Auto-Update of prod DB.
   sync     Synchronize DB state based on the filesystem.
   tasks    Get all possible tasks for a given type and states.
   check    Check the status of tasks.
   cli      Return the equivalent command line entrypoint for tasks.
   script   Generate a shell or slurm script.
   run      Generate a script and run it.
   status   Overview of production.

DESI pipeline control

positional arguments:
  command     Subcommand to run

For instance, desi_pipe top

------------------------------------------------------------------------------
         Task Type        | waiting | ready   | running | done    | fail    |
------------------------------------------------------------------------------
pix                       |        0|    11400|        0|      600|        0|
psf                       |     1710|        0|        0|       90|        0|
psfnight                  |      570|        0|        0|       30|        0|
extract                   |     9690|        0|        0|      510|        0|
fiberflat                 |     1710|        0|        0|       90|        0|
fiberflatnight            |      570|        0|        0|       30|        0|
sky                       |     7980|        0|        0|      420|        0|
starfit                   |     2660|        0|        0|      140|        0|
fluxcalib                 |     7980|        0|        0|      420|        0|
cframe                    |     7980|        0|        0|      420|        0|
spectra                   |     2194|        0|        0|      211|        0|
redshift                  |     2194|        0|        0|      211|        0|
------------------------------------------------------------------------------
 (Use Ctrl-C to Quit)

In brief, Ted and I are very happy about this. We are going to merge as soon as Travis tests pass.