Permalink
Browse files

Implement ``Engine`` and ``Runnable`` abstractions.

``Runnable``s are things such as  Galaxy tools, Galaxy workflows, CWL tools, CWL workflows and ``Engine``s provide operations over ``Runnable``s. There are two Engines right now - cwltool and Galaxy and one command that uses the Engine abstraction (``run`` - formerly ``cwl_run``).

Exit code handling is improved to so fixed exit codes result when unknown runnables are encountered or when a runnable is matched with an incompatible engine. A dependency on ``aenum`` is added to get enable nice enum functionality in planemo to describe runnable types.

The ``Engine`` abstraction implements running Galaxy and CWL tools so far (with either Galaxy or cwltool) but this commit contain a lot of infrastructure for running Galaxy tools and workflows of either type. In particular, workflows will now be uploaded as needed and IDs tracked, they can be executed, and what remains is to collect outputs. The generalized run command implements #449 and #364 - at least in some initial fashion.

Initial infrastructure for a generic test command across engine and runnable types is also present (xref #448). If ``planemo test`` encounters a runnable type other than galaxy_tool and directory - such as a cwl tool, a workflow, etc... it will use a new planemo engine-driven testing procedure. To do this an engine "job" is executed, the results collected, and the results "checked". Currently, Galaxy workflows do not have their results collected and the only "check" available is exact non-File matches and sha1 checksums as used by ``cwltest.py``. A complete suite of checks matching what is available in Galaxy will hopefully be implemented shortly.

This new engine-driven testing produces a structured test data structure that should match what is generated by Galaxy regardless of which engine is used. The upshot of this is that existing test report infrastructure in planemo should be reusable. ``planemo test_report`` and ``planemo share_test`` should work for any ``Engine`` and any ``RunnableType``.

Despite not yet using ``Engine``s infrastructure, the ``serve`` command and related infrastructure for Galaxy has been updated to consume runnables instead of paths. An upshot of this is that serve has been updated to allow serving workflows (either ``.ga`` files directly or Format 2 Workflows through the new gxformat2 library).

In order to serve, run, and test gxformat2 workflows - planemo now depends on the gxformat2 library (https://github.com/jmchilton/gxformat2). The depenedency on bioblend has been constrained to be newer so that workflow invocation client stuff can be used.

The commit also contains general improvements for CWL support - in particular the cwl-runner implementation driven by planemo is much enhanced and now handles (in at least a minimal fashion) all arguments required to be used by the conformance test code. This commit also introduces a test script ``scripts/test_confromance_cwl.bash`` that runs the common workflow language conformance tests against Planemo/Galaxy. Fixes #453.

This commit also includes a variety of improvements to ``run`` and ``serve`` tests.

 - Refactor serve test cases to allow more reuse.
 - Fix profile test to allow rerunning on the same machine more than once.
 - Docstrings.
  • Loading branch information...
jmchilton committed Apr 15, 2016
1 parent 4d29bf1 commit 7be1bf5b3971a43eaa73f483125bfb8cabf1c440
Showing with 2,344 additions and 426 deletions.
  1. +3 −0 .gitignore
  2. +3 −0 Makefile
  3. +41 −2 cwl-runner/cwl-runner
  4. +8 −8 docs/planemo.commands.rst
  5. +0 −8 docs/planemo.cwl.rst
  6. +46 −0 docs/planemo.engine.rst
  7. +16 −0 docs/planemo.galaxy.rst
  8. +10 −0 docs/planemo.rst
  9. +22 −0 docs/planemo.test.rst
  10. +2 −7 planemo/cli.py
  11. +0 −34 planemo/commands/cmd_cwl_run.py
  12. +58 −0 planemo/commands/cmd_run.py
  13. +3 −1 planemo/commands/cmd_serve.py
  14. +24 −6 planemo/commands/cmd_test.py
  15. +3 −1 planemo/commands/cmd_tool_factory.py
  16. +2 −2 planemo/cwl/__init__.py
  17. +0 −125 planemo/cwl/client.py
  18. +89 −27 planemo/cwl/run.py
  19. +14 −0 planemo/engine/__init__.py
  20. +22 −0 planemo/engine/cwltool.py
  21. +50 −0 planemo/engine/factory.py
  22. +42 −0 planemo/engine/galaxy.py
  23. +108 −0 planemo/engine/interface.py
  24. +14 −0 planemo/exit_codes.py
  25. +424 −0 planemo/galaxy/activity.py
  26. +23 −8 planemo/galaxy/api.py
  27. +30 −5 planemo/galaxy/config.py
  28. +16 −9 planemo/galaxy/serve.py
  29. +10 −16 planemo/galaxy/test/actions.py
  30. +5 −53 planemo/galaxy/test/structures.py
  31. +70 −0 planemo/galaxy/workflows.py
  32. +21 −0 planemo/io.py
  33. +5 −0 planemo/network_util.py
  34. +45 −2 planemo/options.py
  35. +425 −0 planemo/runnable.py
  36. +1 −0 planemo/test/__init__.py
  37. +116 −0 planemo/test/results.py
  38. +2 −1 requirements.txt
  39. +14 −0 scripts/test_confromance_cwl.bash
  40. +1 −0 setup.py
  41. +19 −0 tests/data/cat_tool.cwl
  42. +6 −0 tests/data/cat_tool_job.json
  43. +6 −0 tests/data/cat_tool_test.yml
  44. +1 −0 tests/data/hello.txt
  45. +9 −0 tests/data/int_tool.cwl
  46. +1 −0 tests/data/int_tool_job.json
  47. +4 −0 tests/data/int_tool_test.yml
  48. +7 −0 tests/data/randomlines_job_1.json
  49. +87 −0 tests/data/test_workflow_1.ga
  50. +6 −0 tests/data/wf1.gxwf-job.yml
  51. +5 −0 tests/data/wf1.gxwf-test.yml
  52. +13 −0 tests/data/wf1.gxwf.yml
  53. +46 −37 tests/test_cmd_serve.py
  54. +43 −0 tests/test_cmd_test.py
  55. +0 −49 tests/test_cwl_run.py
  56. +48 −0 tests/test_engines.py
  57. +33 −1 tests/test_galaxy_serve.py
  58. +15 −0 tests/test_galaxy_workflow_utils.py
  59. +105 −0 tests/test_run.py
  60. +1 −2 tests/test_shed_download.py
  61. +1 −18 tests/test_shed_upload.py
  62. +78 −0 tests/test_test_checker.py
  63. +22 −4 tests/test_utils.py
@@ -19,8 +19,11 @@ develop-eggs
lib
lib64

# For building cwl-runner package.
cwl-runner/dist
cwl-runner/build
# For running CWL conformance tests.
common-workflow-language

# Installer logs
pip-log.txt
@@ -42,6 +42,9 @@ clean-test: ## remove test and coverage artifacts
rm -f .coverage
rm -fr htmlcov/

install:
python setup.py install && cd cwl-runner && python setup.py install

setup-venv: ## setup a development virutalenv in current directory
if [ ! -d $(VENV) ]; then virtualenv $(VENV); exit; fi;
$(IN_VENV) pip install -r requirements.txt && pip install -r dev-requirements.txt
@@ -1,3 +1,42 @@
#!/bin/sh
#!/bin/bash

planemo cwl_run "$@"
PLANEMO_BIN="${PLANEMO_BIN:-planemo}"
PLANEMO_ARGS="${PLANEMO_ARGS:-}"

VERBOSE_ARG="--verbose"
USE_CONTAINER=0

# Pop args meant for common_startup.sh
while :
do
case "$1" in
--version|--help)
$PLANEMO_BIN "$1"
exit $?
;;
--no-container)
USE_CONTAINER=1
shift
;;
--quiet)
VERBOSE_ARG=""
shift
;;
--basedir*)
BASEDIR=`echo $1 | sed -e 's/^[^=]*=//g'`
#echo `pwd`
#cd "$BASEDIR"
shift
;;
"")
break
;;
*)
PLANEMO_ARGS="$PLANEMO_ARGS $1"
shift
;;
esac
done

# TODO: Do something with USE_CONTAINER
$PLANEMO_BIN $VERBOSE_ARG run $PLANEMO_ARGS
@@ -68,14 +68,6 @@ planemo.commands.cmd_create_gist module
:undoc-members:
:show-inheritance:

planemo.commands.cmd_cwl_run module
-----------------------------------

.. automodule:: planemo.commands.cmd_cwl_run
:members:
:undoc-members:
:show-inheritance:

planemo.commands.cmd_cwl_script module
--------------------------------------

@@ -140,6 +132,14 @@ planemo.commands.cmd_project_init module
:undoc-members:
:show-inheritance:

planemo.commands.cmd_run module
-------------------------------

.. automodule:: planemo.commands.cmd_run
:members:
:undoc-members:
:show-inheritance:

planemo.commands.cmd_serve module
---------------------------------

@@ -4,14 +4,6 @@ planemo.cwl package
Submodules
----------

planemo.cwl.client module
-------------------------

.. automodule:: planemo.cwl.client
:members:
:undoc-members:
:show-inheritance:

planemo.cwl.run module
----------------------

@@ -0,0 +1,46 @@
planemo.engine package
======================

Submodules
----------

planemo.engine.cwltool module
-----------------------------

.. automodule:: planemo.engine.cwltool
:members:
:undoc-members:
:show-inheritance:

planemo.engine.factory module
-----------------------------

.. automodule:: planemo.engine.factory
:members:
:undoc-members:
:show-inheritance:

planemo.engine.galaxy module
----------------------------

.. automodule:: planemo.engine.galaxy
:members:
:undoc-members:
:show-inheritance:

planemo.engine.interface module
-------------------------------

.. automodule:: planemo.engine.interface
:members:
:undoc-members:
:show-inheritance:


Module contents
---------------

.. automodule:: planemo.engine
:members:
:undoc-members:
:show-inheritance:
@@ -11,6 +11,14 @@ Subpackages
Submodules
----------

planemo.galaxy.activity module
------------------------------

.. automodule:: planemo.galaxy.activity
:members:
:undoc-members:
:show-inheritance:

planemo.galaxy.api module
-------------------------

@@ -51,6 +59,14 @@ planemo.galaxy.serve module
:undoc-members:
:show-inheritance:

planemo.galaxy.workflows module
-------------------------------

.. automodule:: planemo.galaxy.workflows
:members:
:undoc-members:
:show-inheritance:


Module contents
---------------
@@ -8,11 +8,13 @@ Subpackages

planemo.commands
planemo.cwl
planemo.engine
planemo.galaxy
planemo.linters
planemo.reports
planemo.shed
planemo.shed2tap
planemo.test
planemo.xml

Submodules
@@ -114,6 +116,14 @@ planemo.options module
:undoc-members:
:show-inheritance:

planemo.runnable module
-----------------------

.. automodule:: planemo.runnable
:members:
:undoc-members:
:show-inheritance:

planemo.shed_lint module
------------------------

@@ -0,0 +1,22 @@
planemo.test package
====================

Submodules
----------

planemo.test.results module
---------------------------

.. automodule:: planemo.test.results
:members:
:undoc-members:
:show-inheritance:


Module contents
---------------

.. automodule:: planemo.test
:members:
:undoc-members:
:show-inheritance:
@@ -11,9 +11,10 @@
OptionSource,
)
from planemo.galaxy import profiles
from planemo.exit_codes import ExitCodeException
from planemo import __version__

PYTHON_2_7_COMMANDS = ["cwl_run", "cwl_script"]
PYTHON_2_7_COMMANDS = ["run", "cwl_script"]
IS_PYTHON_2_7 = sys.version_info[0] == 2 and sys.version_info[1] >= 7


@@ -25,12 +26,6 @@
}


class ExitCodeException(Exception):

def __init__(self, exit_code):
self.exit_code = exit_code


class Context(object):
"""Describe context of Planemo computation.

This file was deleted.

Oops, something went wrong.
@@ -0,0 +1,58 @@
"""Module describing the planemo ``cwl_run`` command."""
from __future__ import print_function

import json

import click

from planemo.cli import command_function
from planemo import options
from planemo.engine import engine_context
from planemo.io import conditionally_captured_io, warn


@click.command('run')
@options.required_tool_arg()
@options.required_job_arg()
@options.galaxy_serve_options()
@options.galaxy_cwl_root_option()
@options.cwl_conformance_test()
@options.run_output_directory_option()
@options.run_output_json_option()
@options.engine_option()
@command_function
def cli(ctx, path, job_path, **kwds):
"""Planemo command for running tools and jobs.
::
% planemo run cat1-tool.cwl cat-job.json
"""
kwds["cwl"] = path.endswith(".cwl")
conformance_test = kwds.get("conformance_test", False)

with conditionally_captured_io(conformance_test):
with engine_context(ctx, **kwds) as engine:
run_result = engine.run(path, job_path)

if not run_result.was_successful:
warn("Run failed [%s]" % str(run_result))
ctx.exit(1)

if conformance_test:
if hasattr(run_result, "cwl_command_state"):
command_state = run_result.cwl_command_state
dumped_json = json.dumps(command_state)
if hasattr(run_result, "galaxy_paths"):
for (local_path, galaxy_path) in run_result.galaxy_paths:
dumped_json = dumped_json.replace(galaxy_path, local_path)
print(dumped_json)
else:
outputs_dict = run_result.outputs_dict
print(outputs_dict)
output_json = kwds.get("output_json", None)
if output_json:
with open(output_json, "w") as f:
json.dump(outputs_dict, f)

return 0
@@ -2,6 +2,7 @@
import click

from planemo.cli import command_function
from planemo.runnable import for_paths
from planemo.galaxy import galaxy_serve
from planemo import options

@@ -35,4 +36,5 @@ def cli(ctx, paths, **kwds):
proof yet so please careful and do not try this against production Galaxy
instances.
"""
galaxy_serve(ctx, paths, **kwds)
runnables = for_paths(paths)
galaxy_serve(ctx, runnables, **kwds)
Oops, something went wrong.

0 comments on commit 7be1bf5

Please sign in to comment.