doc page for miniwdl check

WDL/CLI.py

@@ -40,22 +40,7 @@
 def main(args=None):
     sys.setrecursionlimit(1_000_000)  # permit as much call stack depth as OS can give us
 
-    parser = ArgumentParser("miniwdl")
-    parser.add_argument(
-        "--version",
-        nargs=0,
-        action=PipVersionAction,
-        help="show miniwdl package version information",
-    )
-    subparsers = parser.add_subparsers()
-    subparsers.required = True
-    subparsers.dest = "command"
-    fill_common(fill_check_subparser(subparsers))
-    fill_common(fill_cromwell_subparser(subparsers), path=False)  # FIXME path issue #131
-    fill_common(fill_run_subparser(subparsers))
-    fill_common(fill_run_self_test_subparser(subparsers))
-    fill_common(fill_localize_subparser(subparsers))
-
+    parser = create_arg_parser()
     argcomplete.autocomplete(parser)
 
     replace_COLUMNS = os.environ.get("COLUMNS", None)
@@ -98,6 +83,25 @@ def main(args=None):
     sys.exit(0)
 
 
+def create_arg_parser():
+    parser = ArgumentParser("miniwdl")
+    parser.add_argument(
+        "--version",
+        nargs=0,
+        action=PipVersionAction,
+        help="show miniwdl package version information",
+    )
+    subparsers = parser.add_subparsers()
+    subparsers.required = True
+    subparsers.dest = "command"
+    fill_common(fill_check_subparser(subparsers))
+    fill_common(fill_cromwell_subparser(subparsers), path=False)  # FIXME path issue #131
+    fill_common(fill_run_subparser(subparsers))
+    fill_common(fill_run_self_test_subparser(subparsers))
+    fill_common(fill_localize_subparser(subparsers))
+    return parser
+
+
 class PipVersionAction(Action):
     def __call__(self, parser, namespace, values, option_string=None):
         try:
@@ -136,7 +140,8 @@ def fill_common(subparser, path=True):
             action="append",
             help="local directory to search for imports",
         )
-    subparser.add_argument(
+    group = subparser.add_argument_group("debugging")
+    group.add_argument(
         "--debug", action="store_true", help="maximally verbose logging & exception tracebacks"
     )
 
@@ -153,7 +158,7 @@ def fill_check_subparser(subparsers):
         formatter_class=RawDescriptionHelpFormatter,
     )
     check_parser.add_argument(
-        "uri", metavar="URI", type=str, nargs="+", help="WDL document filename/URI"
+        "uri", metavar="WDL_URI", type=str, nargs="+", help="WDL document filename/URI"
     )
     check_parser.add_argument(
         "--strict",

WDL/__init__.py

@@ -2,7 +2,7 @@
 `miniwdl <https://github.com/chanzuckerberg/miniwdl/>`_ is a developer toolkit and local runner for
 the bioinformatics-focused `Workflow Description Language (WDL) <http://openwdl.org/>`_. This
 documentation covers the Python3 ``WDL`` package facilitating parsing & static analysis of WDL
-documents.
+documents. Simply ``import WDL`` once miniwdl has been installed.
 
 * `GitHub repo <https://github.com/chanzuckerberg/miniwdl/>`_ for installation and further background
 * `Codelabs <https://miniwdl.readthedocs.io/en/latest/WDL.html#python-codelabs>`_ on using this package

docs/WDL.rst

@@ -3,8 +3,8 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-WDL Python package
-==================
+Python ``WDL`` package
+======================
 
 .. contents::
 

docs/check.rst

@@ -0,0 +1,106 @@
+``miniwdl check``
+=================
+
+To aid the workflow development cycle, miniwdl includes a code quality checker which statically analyzes WDL source code to generate warnings and suggestions. Take for example this valid but poorly-written WDL task:
+
+.. code-block:: bash
+
+   $ cat << 'EOF' > check_demo.wdl
+     task t {
+         String s = i
+         Int? i
+         command <<<
+            if [ ! -n ~{s} ]; then
+                echo Empty
+            fi
+         >>>
+         output {
+             String t = read_string(stdout())
+         }
+     }
+   EOF
+
+Run this through ``miniwdl check``:
+
+.. code-block:: none
+
+   $ miniwdl check check_demo.wdl
+   check_demo.wdl
+        (Ln 1, Col 1) MissingVersion, document should declare WDL version; draft-2 assumed
+        task t
+            (Ln 2, Col 10) StringCoercion, String s = :Int?:
+            (Ln 2, Col 10) UnusedDeclaration, nothing references String s
+            (Ln 2, Col 21) ForwardReference, reference to i precedes its declaration
+            (Ln 10, Col 14) NameCollision, declaration of 't' collides with a task name
+
+miniwdl parsed the document successfully to produce this outline, but noted several issues within. First, we forgot to specify the WDL language version by starting the file with ``version 1.0`` or ``version development``, causing miniwdl to assume the outdated draft-2 dialect (as required by the WDL specification). This in turn causes a more serious problem: WDL draft-2 didn't yet support the ``~{expr}`` interpolation syntax, so it goes unrecognized here, leaving incorrect command logic and the WDL value ``s`` unused. This pitfall (a common one!) illustrates how the "lint" warnings, while often stylistic, can indicate critical errors.
+
+If your system has `ShellCheck <https://www.shellcheck.net/>`_ installed, ``miniwdl check`` automatically runs it on each task command script and reports any findings, in this case:
+
+.. code-block:: none
+
+   (Ln 5, Col 17) CommandShellCheck, SC2236 Use -z instead of ! -n.
+
+The ``miniwdl check`` process succeeds (zero exit status code) so long as the WDL document can be parsed and type-checked, even if lint or ShellCheck warnings are reported. With ``--strict``, lint warnings as well as parse/type errors lead to a non-zero exit status.
+
+Suppressing warnings
+--------------------
+
+Individual warnings can be suppressed by a WDL comment containing ``!WarningName`` on the same line or the following line, for example:
+
+.. code-block:: bash
+
+   $ cat << 'EOF' > check_demo2.wdl
+     task t {
+         String s = i  # !ForwardReference !StringCoercion
+         Int? i
+         command <<<
+            if [ ! -n ~{s} ]; then
+                echo Empty
+            fi
+         >>>
+         output {
+             String t = read_string(stdout())
+             # Meant to do that: !NameCollision
+         }
+     }
+   EOF
+   $ miniwdl check check_demo2.wdl
+   check_demo2.wdl
+       (Ln 1, Col 1) MissingVersion, document should declare WDL version; draft-2 assumed
+       task t
+           (Ln 5, Col 17) CommandShellCheck, SC2236 Use -z instead of ! -n.
+           (Ln 2, Col 10) UnusedDeclaration, nothing references String s
+
+ShellCheck warnings can be suppressed using `that tool's own convention <https://github.com/koalaman/shellcheck/wiki/Ignore>`_.
+
+Pre-commit hook
+---------------
+
+In a git repository with WDL workflows, you can use `pre-commit <https://pre-commit.com/>`_  with ``miniwdl check`` with by entering into ``.pre-commit-config.yaml``:
+
+.. code-block:: yaml
+
+   repos:
+   - repo: local
+     hooks:
+     - id: miniwdl-check
+       name: miniwdl check
+       language: system
+       files: ".+\\.wdl"
+       verbose: true
+       entry: miniwdl
+       args: [check]
+
+Then try ``pre-commit run --all-files`` or install git hooks according to its procedure; add ``--strict`` to args if desired.
+
+Command line
+------------
+
+.. argparse::
+   :module: WDL.CLI
+   :func: create_arg_parser
+   :prog: miniwdl
+   :path: check
+   :nodescription:
+   :nodefaultconst:

docs/conf.py

@@ -42,7 +42,9 @@
     'sphinx.ext.autodoc',
     'sphinx.ext.doctest',
     'sphinx.ext.viewcode',
-    'sphinx.ext.inheritance_diagram'
+    'sphinx.ext.inheritance_diagram',
+    'recommonmark',
+    'sphinxarg.ext',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -162,11 +164,3 @@
 # -- Extension configuration -------------------------------------------------
 
 autodoc_member_order = 'bysource'
-
-from recommonmark.parser import CommonMarkParser
-
-source_parsers = {
-    '.md': CommonMarkParser,
-}
-
-source_suffix = ['.rst', '.md']

docs/getting_started.md

@@ -160,30 +160,7 @@ Individual tasks and sub-workflows run in their own nested subdirectories, each
 
 ## Next steps
 
-To aid the workflow development cycle, miniwdl includes a code quality checker which statically analyzes WDL source code to generate warnings and suggestions:
-
-```
-$ miniwdl check skylab/library/tasks/ZarrUtils.wdl
-ZarrUtils.wdl
-    task OptimusZarrConversion
-        (Ln 110, Col 6) CommandShellCheck, SC2006 Use $(..) instead of legacy `..`.
-        (Ln 113, Col 9) CommandShellCheck, SC2006 Use $(..) instead of legacy `..`.
-        (Ln 113, Col 15) CommandShellCheck, SC2086 Double quote to prevent globbing and word splitting.
-        (Ln 114, Col 10) CommandShellCheck, SC2086 Double quote to prevent globbing and word splitting.
-        (Ln 114, Col 21) CommandShellCheck, SC2086 Double quote to prevent globbing and word splitting.
-    task OptimusZarrToLoom
-        (Ln 139, Col 5) UnusedDeclaration, nothing references Int cpu
-    task SmartSeq2ZarrConversion
-        (Ln 37, Col 6) CommandShellCheck, SC2006 Use $(..) instead of legacy `..`.
-        (Ln 40, Col 9) CommandShellCheck, SC2006 Use $(..) instead of legacy `..`.
-        (Ln 40, Col 15) CommandShellCheck, SC2086 Double quote to prevent globbing and word splitting.
-        (Ln 41, Col 10) CommandShellCheck, SC2086 Double quote to prevent globbing and word splitting.
-        (Ln 41, Col 21) CommandShellCheck, SC2086 Double quote to prevent globbing and word splitting.
-```
-
-Here miniwdl points out an unused declaration in one task, and (if [ShellCheck](https://www.shellcheck.net/) is installed) several suggestions for the embedded shell commands.
-
-Installing miniwdl also makes available a `WDL` package for Python 3.6+, providing programmatic access to miniwdl's WDL parser and other functionality; its documentation follows here.
+The following pages document features, configuration, and optimization for `miniwdl run`. To aid the workflow development cycle, miniwdl also includes a static code quality checker, `miniwdl check`. Lastly, installing miniwdl makes available a Python `WDL` package, providing programmatic access to miniwdl's WDL parser and runtime.
 
 ### Links
 

docs/index.rst

@@ -8,8 +8,10 @@ the bioinformatics-focused `Workflow Description Language (WDL) <http://openwdl.
    :maxdepth: 2
 
    getting_started.md
+   runner_cli.rst
    runner_reference.md
    runner_advanced.md
+   check.md
    WDL.rst
 
 Links

docs/runner_advanced.md

@@ -1,4 +1,4 @@
-# Runner advanced guidelines
+# `miniwdl run` optimization
 
 Follow these guidelines for optimal performance and reliability from `miniwdl run`.
 

docs/runner_cli.rst

@@ -0,0 +1,9 @@
+``miniwdl run`` command line
+============================
+
+.. argparse::
+   :module: WDL.CLI
+   :func: create_arg_parser
+   :prog: miniwdl
+   :path: run
+   :nodefaultconst:

docs/runner_reference.md

@@ -1,4 +1,4 @@
-# Runner reference
+# `miniwdl run` reference
 
 ## I/O and run directory structure
 

requirements.dev.txt

@@ -17,3 +17,4 @@ pytest
 pytest-cov
 pytest-xdist
 recommonmark
+sphinx-argparse