Merge pull request #68 from epigen/dev

Last updates for first 0.5-rc1 release

.travis.yml

@@ -1,6 +1,9 @@
 language: python
 python:
   - "2.7"
+os:
+  - linux
+  - osx
 before_install:
   - pip install -U pip
 install:
@@ -10,7 +13,5 @@ install:
 script: pytest
 branches:
   only:
-  - features_0.5
   - dev
   - master
-

doc/source/changelog.rst

@@ -1,6 +1,29 @@
 Changelog
 ******************************
 
+- **v0.5** (*2017-03-01*):
+
+  - New
+
+    - Add new looper version tracking, with `--version` and `-V` options and printing version at runtime
+
+    - Add support for asterisks in file paths
+
+    - Add support for multiple pipeline directories in priority order
+
+    - Revamp of messages make more intuitive output
+
+    - Colorize output
+
+    - Complete rehaul of logging and test infrastructure, using logging and pytest packages
+
+  - Fixes
+
+    - Removes pipelines_dir requirement for models, making it useful outside looper
+
+    - Small bug fixes related to `all_input_files` and `required_input_files` attributes
+
+
 - **v0.4** (*2017-01-12*):
 
   - New
@@ -13,6 +36,8 @@ Changelog
 
     - New command (``looper destroy``) to remove all output of a project
 
+    - New command (``looper clean``) to remove intermediate files of a project flagged for deletion
+
     - Support for portable and pipeline-independent allocation of computing resources with Looperenv.
 
   - Fixes

doc/source/tutorials.rst

@@ -27,7 +27,9 @@ Now you can run this project with looper! Just use ``looper run``:
 	looper run microtest/config/microtest_config.yaml
 
 
-If the looper executable isn't in your path, check out the :doc:`FAQ <faq>`.
+.. HINT::
+
+	If the looper executable isn't in your path, add it with export PATH=~/.local/bin:$PATH -- check out the :doc:`FAQ <faq>`.
 
 Pipeline outputs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -43,62 +45,13 @@ The sample-specific output of each pipeline type varies and is described in :doc
 
 To use pre-made pipelines with your project, all you have to do is :doc:`define your project <define-your-project>` using looper's standard format. To link your own, custom built pipelines, you can :doc:`connect your pipeline to looper <connecting-pipelines>`.
 
-
-
-.. rubric:: Footnotes
-
-.. [1] This variable can also be specified in the ``results_subdir`` variable under the ``paths`` section of the project config file
-.. [2] This variable can also be specified in the ``submission_subdir`` variable under the ``paths`` section of the project config file
-
-
-First, install looper and pypiper (since our tutorial uses pypiper pipelines):
-
-
-.. code:: bash
-
-	pip install --user https://github.com/epigen/looper/zipball/master
-	pip install --user https://github.com/epigen/pypiper/zipball/master
-
-
-Now, you will need to grab a project to run, and some pipelines to run on it. We have a functional working project example and an open source pipeline repository on github.
-
-
-.. code:: bash
-
-	git clone https://github.com/epigen/microtest.git
-	git clone https://github.com/epigen/open_pipelines.git
-
-
-Now you can run this project with looper! Just use ``looper run``:
-
-.. code:: bash
-
-	looper run microtest/config/microtest_config.tutorial.yaml
-
-.. HINT::
-
-	If the looper executable isn't in your path, add it with export PATH=~/.local/bin:$PATH -- check out the :doc:`FAQ <faq>`.
-
-
 In this example, we just ran one example sample (an amplicon sequencing library) through a pipeline that processes amplicon data (to determine percentage of indels in amplicon).
 
 From here to running hundreds of samples of various sample types is virtually the same effort!
 
 
-Pipeline outputs
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-Outputs of pipeline runs will be under the directory specified in the ``metadata.output_dir`` variable in the project config file. This is usually the name of the project being run; in our example, it's `$HOME/microtest`.
-
-Inside there will be two directories:
-
--  ``results_pipeline`` [1]_ - a directory containing one directory with the output of the pipelines, for each sample.
--  ``submissions`` [2]_ - which holds yaml representations of the samples and log files of the submited jobs.
-
-
-To use pre-made pipelines with your project, all you have to do is :doc:`define your project <define-your-project>` using looper's standard format. To link your own, custom built pipelines, you can :doc:`connect your pipeline to looper <connecting-pipelines>`.
-
 
 .. rubric:: Footnotes
 
-.. [1] This variable can also be specified in the ``results_subdir`` variable under the ``metadata`` section of the project config file
-.. [2] This variable can also be specified in the ``submission_subdir`` variable under the ``metadata`` section of the project config file
+.. [1] This variable can also be specified in the ``results_subdir`` variable under the ``paths`` section of the project config file
+.. [2] This variable can also be specified in the ``submission_subdir`` variable under the ``paths`` section of the project config file

looper/__init__.py

@@ -41,6 +41,9 @@ def setup_looper_logger(level, additional_locations=None, devmode=False):
     :return logging.Logger: project-root logger
     """
 
+    logging.addLevelName(0, "EVERYTHING")
+    logging.addLevelName(5, "VERY_FINE")
+
     fmt = DEV_LOGGING_FMT if devmode else DEFAULT_LOGGING_FMT
 
     # Establish the logger.
@@ -83,7 +86,7 @@ def setup_looper_logger(level, additional_locations=None, devmode=False):
     for loc in where:
         if isinstance(loc, str):
             # File destination
-            dirpath = os.path.dirname(loc)
+            dirpath = os.path.abspath(os.path.dirname(loc))
             if not os.path.exists(dirpath):
                 os.makedirs(dirpath)
             handler_type = logging.FileHandler
@@ -95,7 +98,12 @@ def setup_looper_logger(level, additional_locations=None, devmode=False):
             logging.info("{} as logs destination appears to be neither "
                          "a filepath nor a stream.".format(loc))
             continue
-        handler = handler_type(loc)
+
+        if handler_type is logging.FileHandler:
+            handler = handler_type(loc, mode='w')
+        else:
+            handler = handler_type(loc)
+
         handler.setLevel(level)
         handler.setFormatter(formatter)
         LOOPER_LOGGER.addHandler(handler)

looper/_version.py

@@ -1 +1 @@
-__version__ = "0.5.0"
+__version__ = "0.5.0-rc1"

looper/looper.py

@@ -69,7 +69,6 @@ def parse_arguments():
                         choices=range(len(_LEVEL_BY_VERBOSITY)),
                         help=argparse.SUPPRESS)
     parser.add_argument("--logging-level",
-                        default=LOGGING_LEVEL,
                         help=argparse.SUPPRESS)
     parser.add_argument("--dbg",
                         action="store_true",
@@ -153,13 +152,13 @@ def parse_arguments():
     # Set the logging level.
     if args.dbg:
         # Debug mode takes precedence and will listen for all messages.
-        level = logging.DEBUG
+        level = args.logging_level or logging.DEBUG
     elif args.verbosity is not None:
         # Verbosity-framed specification trumps logging_level.
         level = _LEVEL_BY_VERBOSITY[args.verbosity]
     else:
         # Normally, we're not in debug mode, and there's not verbosity.
-        level = args.logging_level
+        level = LOGGING_LEVEL
 
     # Establish the project-root logger and attach one for this module.
     setup_looper_logger(level=level,
@@ -204,6 +203,7 @@ def run(prj, args, remaining_args, interface_manager):
         _LOGGER.info(_COUNTER.show(sample.sample_name, sample.library))
 
         pipeline_outfolder = os.path.join(prj.metadata.results_subdir, sample.sample_name)
+        _LOGGER.debug("Pipeline output folder: '%s'", pipeline_outfolder)
         fail_message = ""
 
         # Don't submit samples with duplicate names
@@ -337,7 +337,7 @@ def run(prj, args, remaining_args, interface_manager):
                     cmd += " -M " + submit_settings["mem"]
 
             # Add the command string and job name to the submit_settings object
-            submit_settings["JOBNAME"] = sample.sample_name + "_" + pipeline_job
+            submit_settings["JOBNAME"] = sample.sample_name + "_" + pl_id
             submit_settings["CODE"] = cmd
 
             # Submit job!
@@ -734,8 +734,16 @@ def main():
             prj.set_compute(args.compute)
 
         # TODO split here, spawning separate run process for each pipelines directory in project metadata pipelines directory.
-        pipedirs = prj.metadata.pipelines_dir
-        _LOGGER.debug("Pipelines dirpath(s): {}".format(pipedirs))
+        try:
+            pipedirs = prj.metadata.pipelines_dir
+            _LOGGER.info("Pipelines path(s): {}".format(pipedirs))
+        except AttributeError:
+            _LOGGER.error("Looper requires a metadata.pipelines_dir")
+            raise
+
+        if len(pipedirs) == 0:
+            _LOGGER.error("Looper requires a metadata.pipelines_dir")   
+            raise AttributeError         
 
         interface_manager = InterfaceManager(prj.metadata.pipelines_dir)
         try: