misc:

- include feature of abstracting over resource managers on README
- replace downloads/month with downloads/week badge
- add total num of downloads badge
- remove unused imports in test_common.py and test_config.py
- tweaks for cmd_run docstrings
- docs tweaks for section on container runtimes, resource managers

README.md

@@ -1,6 +1,7 @@
 # <img src="https://raw.githubusercontent.com/systemslab/popper/57f7a89bed6ff3e4d62ea2a5683ae28e3251931e/docs/figures/popper_logo_just_jug.png" width="64" valign="middle" alt="Popper"/> Popper
 
-[![Downloads](https://pepy.tech/badge/popper/month)](https://pepy.tech/project/popper)
+[![Downloads](https://pepy.tech/badge/popper)](https://pepy.tech/project/popper)
+[![Downloads Per Week](https://pepy.tech/badge/popper/week)](https://pepy.tech/project/popper)
 [![Build Status](https://travis-ci.org/systemslab/popper.svg?branch=master)](https://travis-ci.org/systemslab/popper)
 [![codecov](https://codecov.io/gh/systemslab/popper/branch/master/graph/badge.svg)](https://codecov.io/gh/systemslab/popper)
 [![Join the chat at https://gitter.im/systemslab/popper](https://badges.gitter.im/systemslab/popper.svg)](https://gitter.im/falsifiable-us/popper?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
@@ -42,8 +43,14 @@ The high-level goals of this project are to provide:
     for workflows instead of services.
   * **An abstraction over container runtimes**. In addition to Docker, 
     Popper can seamlessly execute workflows in other runtimes by 
-    interacting with distinct container engines. We currently support 
-    [Singularity][sylabs] and are working on adding [Podman][podman].
+    interacting with distinct container engines. Popper currently 
+    supports [Singularity][sylabs] and we are working on adding 
+    [Podman][podman].
+  * **Run on resource managers**. Popper can also execute workflows on 
+    a variety of resource managers and schedulers such as Kubernetes 
+    and SLURM, without requiring any modifications to a workflow YAML 
+    file. We currently support SLURM and are working on adding support 
+    for Kubernetes.
   * **Continuous integration**. Generate configuration files for 
     distinct CI services, allowing users to run the exact same 
     workflows they run locally on Travis, Jenkins, Gitlab, Circle and 

cli/popper/commands/cmd_run.py

@@ -114,28 +114,27 @@
     default=os.getcwd()
 )
 @click.option(
-    '-c', '--conf', help='Runtime configuration options.', required=False
+    '-c',
+    '--conf',
+    help='Path to file with configuration options.',
+    required=False
 )
 @pass_context
 def cli(ctx, step, wfile, debug, dry_run, log_file, quiet, reuse,
         engine, resource_manager, skip, skip_pull, skip_clone,
         substitution, allow_loose, with_dependencies, workspace, conf):
     """Runs a Popper workflow. Only executes STEP if given.
 
-    This command allows specifying the engine and the resource manager
-    in two different ways.
-
-    * Using the `--engine/-e` option and `--resource-manager/-r` option.
-
-    * Through a configuration file specified with the `--conf/-c` option.
-
-    NOTE:
-
-    1. If none of the above are given, popper uses docker as the
-       default engine and host as the default resource manager.
+    To specify a container engine to use other than docker, use the --engine/-e
+    flag. For executing on a resource manager such as SLURM or Kubernetes, use
+    the --resource-manager/-r flag. Alternatively, a configuration file can be
+    given (--conf flag) that can specify container options, resource manager
+    options, or both (see "Workflow Syntax and Execution Runtime" section of
+    the Popper documentation for more).
 
-    2. If the engine or resource manager is specified through CLI and
-       config file both, CLI is given preference over config file.
+    If the container engine (-e) or resource manager (-r) are specified with a
+    flag and a configuration file is given as well, the values passed via the
+    flags are given preference over those contained in the configuration file.
     """
     # set the logging levels.
     level = 'STEP_INFO'

cli/popper/runner_host.py

@@ -185,7 +185,8 @@ def _create_container(self, cid, step):
 
         if build:
             log.info(
-                f'[{step["name"]}] docker build {img}:{tag} {os.path.dirname(dockerfile)}')
+                f'[{step["name"]}] docker build {img}:{tag} '
+                f'{os.path.dirname(dockerfile)}')
             if not self._config.dry_run:
                 self._d.images.build(path=dockerfile, tag=f'{img}:{tag}',
                                      rm=True, pull=True)
@@ -199,8 +200,16 @@ def _create_container(self, cid, step):
 
         container_args = self._get_container_kwargs(step, f'{img}:{tag}', cid)
 
-        log.info(f'[{step["name"]}] docker create {container_args} {img}:{tag}')
+        msg = f'[{step["name"]}] docker create name={cid}'
+        msg += f' image={container_args["image"]}'
+        if container_args["entrypoint"]:
+            msg += f' entrypoint={container_args["entrypoint"]}'
+        if container_args["command"]:
+            msg += f' command={container_args["command"]}'
+        log.info(msg)
+
         container = self._d.containers.create(**container_args)
+
         return container
 
     def _get_container_kwargs(self, step, img, name):

cli/test/test_common.py

@@ -1,14 +1,8 @@
 import os
-import sys
-import shutil
-import contextlib
-import subprocess
 import tempfile
 import git
 import unittest
 
-FIXDIR = f'{os.path.dirname(os.path.realpath(__file__))}/fixtures'
-
 
 class PopperTest(unittest.TestCase):
 

cli/test/test_config.py

@@ -3,11 +3,8 @@
 
 from popper.config import PopperConfig
 from popper.cli import log
-from test_common import PopperTest
 
 
-FIXDIR = f'{os.path.dirname(os.path.realpath(__file__))}/fixtures'
-
 class TestPopperConfig(unittest.TestCase):
     default_args = {
         'skip_clone': False,

docs/sections/cn_workflows.md

@@ -199,19 +199,25 @@ the exit code to set the workflow execution status, which can be
 
 ## Container Engines
 
-By default, steps in Popper workflows run in Docker using the host machine 
-as the resource manager (see [next section](#resource-managers) on running on other resource managers). 
-In addition, Popper can execute workflows in other runtimes by interacting with 
-their corresponding container engines. An `--engine <engine>` flag for 
-the `popper run` is used to invoke alternative engines, where 
-`<engine>` is one of the supported engines. When no value for this 
-flag is given, Popper executes workflows in Docker. Below we briefly 
-describe each container engine supported (besides Docker), and lastly 
-describe how to customize their configuration.
+By default, Popper workflows run in Docker on the machine where 
+`popper run` is being executed (i.e. the host machine). This section 
+describes how to execute in other container engines. See [next 
+section](#resource-managers) for information on how to run workflows 
+on resource managers  such as SLURM and Kubernetes.
 
-### Supported engines
+To run workflows on other container engines, an `--engine <engine>` 
+flag for the `popper run` command can be given, where `<engine>` is 
+one of the supported ones. When no value for this flag is given, 
+Popper executes workflows in Docker. Below we briefly describe each 
+container engine supported, and lastly describe how to pass 
+engine-specific configuration options via the `--conf` flag.
 
-#### Singularity
+### Docker
+
+Docker is the default engine used by the `popper run`. All the 
+container configuration for the docker engine is supported by Popper.
+
+### Singularity
 
 Popper can execute a workflow in systems where Singularity 3.2+ is 
 available. To execute a workflow in Singularity containers:
@@ -220,12 +226,12 @@ available. To execute a workflow in Singularity containers:
 popper run --engine singularity
 ```
 
-##### Limitations
+#### Limitations
 
   * The use of `ARG` in `Dockerfile`s is not supported by Singularity.
   * The `--reuse` flag of the `popper run` command is not supported.
 
-#### Vagrant
+### Vagrant
 
 While technically not a container engine, executing workflows inside a 
 VM allows users to run workflows in machines where a container engine 
@@ -240,13 +246,13 @@ popper run --engine vagrant
 
 [vagrant]: https://vagrantup.com/
 
-##### Limitations
+#### Limitations
 
 Only one workflow can be executed at the time in Vagrant runtime, 
 since popper assumes that there is only one VM running at any given 
 point in time.
 
-#### Host
+### Host
 
 There are situations where a container runtime is not available and 
 cannot be installed. In these cases, a step can be executed directly 
@@ -295,16 +301,14 @@ question (see [here][engconf] for more).
 ## Resource Managers
 
 Popper can execute steps in a workflow through other resource managers
-like `slurm` besides the host machine. The resource manager can be specified 
-either through the  `--resource-manager/-r` option or through the config file.
+like SLURM besides the host machine. The resource manager can be specified 
+either through the `--resource-manager/-r` option or through the config file.
 If neither of them are provided, the steps are run in the host machine 
 by default. 
 
-### Supported resource managers
+### SLURM
 
-#### Slurm
-
-Popper workflows can be run on [HPC](https://en.wikipedia.org/wiki/HPC) (Multi-Node environments) 
+Popper workflows can run on [HPC](https://en.wikipedia.org/wiki/HPC) (Multi-Node environments) 
 using [Slurm](https://slurm.schedmd.com/overview.html) as the underlying resource manager to distribute the execution of a step to
 several nodes. You can get started with running Popper workflows through Slurm by following the example below.
 
@@ -348,6 +352,7 @@ resource_manager:
 ```
 
 Now, we execute `popper run` with this config file as follows:
+
 ```bash
 popper run -f sample.yml -c config.yml
 ```