Implement singularity-compose.yml override configuration feature (#48)

* implement deep_merge functionally
* implement some changes suggested during code review
* add tests for the config override feature
* good old black formatting that I keep forgetting :)
* add docs
* adding support for singularity-compose check

this will add jsonschema as an optional dependency to check the validity of a
singularity-compose.yml file. We will eventually extend this to check (and preview)
combined files.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

* implement deep_merge functionally
* implement some changes suggested during code review
* add tests for the config override feature
* tweak check group so it takes into account multiple files instead
* implement preview option
* update changelog and add docs
* Apply suggestions from code review

Co-authored-by: Vanessasaurus <814322+vsoch@users.noreply.github.com>

* move deep_merge stuff to a brand new module
* black formatting
* remove extra bullet point
* implement code review changes
* move config functions into proper module

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

and moving validate.py into its own file so we do not always import
* use bot.exit over print and sys.exit  (which combines the two)

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: Vanessasaurus <814322+vsoch@users.noreply.github.com>

.github/workflows/main.yml

@@ -14,12 +14,13 @@ jobs:
       - uses: actions/checkout@v2
 
       - name: Setup black environment
-        run: conda create --quiet --name black black pyflakes
+        run: conda create --quiet --name black pyflakes
 
       - name: Check formatting with black
         run: |
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate black
+          pip install black==21.6b0
           black --check scompose
 
       - name: Check imports with pyflakes
@@ -28,4 +29,4 @@ jobs:
           source activate black
           pyflakes scompose/utils
           # Will have some issues
-          pyflakes scompose/client scompose/project || true
+          pyflakes scompose/client scompose/project scompose/config || true

CHANGELOG.md

@@ -14,6 +14,10 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pypi.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-compose/tree/master) (0.0.x)
+
+ - adding jsonschema validation and check command (0.0.12)
+   - implement configuration override feature
+   - implement `--preview` argument for the `check` command 
  - add network->enable option on composer file (0.1.11)
  - add network->allocate_ip option on composer file (0.1.10)
  - version 2.0 of the spec with added fakeroot network, start, exec, and run options (0.1.0)

docs/commands.md

@@ -3,7 +3,48 @@
 The following commands are currently supported. Remember, you must be in the 
 present working directory of the compose file to reference the correct instances.
 
-## Build
+## check
+
+To do a sanity check of your singularity-compose.yml, you can use `singularity-compose check`
+
+```bash
+$ singularity-compose check
+singularity-compose.yml is valid.
+
+$ singularity-compose -f singularity-compose.yml \ 
+          -f singularity-compose.override.yml check 
+singularity-compose.yml is valid.
+singularity-compose.override.yml is valid.
+```
+
+To view the combined compose files you can use `--preview`.
+
+```bash
+$ singularity-compose -f singularity-compose.yml \ 
+          -f singularity-compose.override.yml check  --preview
+
+version: '2.0'
+instances:
+  cvatdb:
+    start:
+      options:
+      - containall
+    network:
+      enable: false
+    volumes:
+    - ./recipes/postgres/env.sh:/.singularity.d/env/env.sh
+    - ./volumes/postgres/conf:/opt/bitnami/postgresql/conf
+    - ./volumes/postgres/tmp:/opt/bitnami/postgresql/tmp
+    - /home/vagrant/postgres_data:/bitnami/postgresql
+    build:
+      context: .
+      recipe: ./recipes/postgres/main.def
+      options:
+      - fakeroot
+
+```
+
+## build
 
 Build will either build a container recipe, or pull a container to the
 instance folder. In both cases, it's named after the instance so we can
@@ -21,7 +62,7 @@ If the build requires sudo (if you've defined sections in the config that warran
 setting up networking with sudo) the build will instead give you an instruction
 to run with sudo.
 
-## Up
+## up
 
 If you want to both build and bring them up, you can use "up." Note that for
 builds that require sudo, this will still stop and ask you to build with sudo.
@@ -52,7 +93,7 @@ $ singularity-compose up --no-resolv
 Creating app
 ```
 
-## Create
+## create
 
 Given that you have built your containers with `singularity-compose build`,
 you can create your instances as follows:
@@ -93,7 +134,7 @@ INSTANCES  NAME PID     IMAGE
 3         nginx	6543	nginx.sif
 ```
 
-## Shell
+## shell
 
 It's sometimes helpful to peek inside a running instance, either to look at permissions,
 inspect binds, or manually test running something.
@@ -104,7 +145,7 @@ $ singularity-compose shell app
 Singularity app.sif:~/Documents/Dropbox/Code/singularity/singularity-compose-example> 
 ```
 
-## Exec
+## exec
 
 You can easily execute a command to a running instance:
 
@@ -134,7 +175,7 @@ usr
 var
 ```
 
-## Run
+## run
 
 If a container has a `%runscript` section (or a Docker entrypoint/cmd that 
 was converted to one), you can run that script easily:
@@ -147,7 +188,7 @@ If your container didn't have any kind of runscript, the startscript
 will be used instead.
 
 
-## Down
+## down
 
 You can bring one or more instances down (meaning, stopping them) by doing:
 
@@ -172,7 +213,7 @@ in order to kill instances after the specified number of seconds:
 singularity-compose down -t 100
 ```
 
-## Logs
+## logs
 
 You can of course view logs for all instances, or just specific named ones:
 
@@ -190,7 +231,7 @@ nginx: [emerg] host not found in upstream "uwsgi" in /etc/nginx/conf.d/default.c
 nginx: [emerg] host not found in upstream "uwsgi" in /etc/nginx/conf.d/default.conf:22
 ```
 
-## Config
+## config
 
 You can load and validate the configuration file (singularity-compose.yml) and
 print it to the screen as follows:
@@ -245,4 +286,111 @@ $ singularity-compose config
 }
 ```
 
+# Global arguments
+
+The following arguments are supported for all commands available.
+
+## debug
+
+Set logging verbosity to debug.
+
+```bash
+$ singularity-compose --debug version
+```
+
+This is equivalent to passing `--log-level=DEBUG` to the CLI.
+
+```bash
+$ singularity-compose --log-level='DEBUG' version
+```
+
+## log_level
+
+Change logging verbosity. Accepted values are: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`
+
+```bash
+$ singularity-compose --log-level='INFO' version
+```
+
+## file
+
+Specify the location of a Compose configuration file
+
+Default value: `singularity-compose.yml`
+
+Aliases `--file`, `-f`. 
+
+You can supply multiple `-f` configuration files. When you supply multiple files, `singularity-compose`
+ combines them into a single configuration. It builds the configuration in the order you supply the
+files. Subsequent files override and add to their predecessors.
+
+For example consider this command:
+
+```bash
+$ singularity-compose -f singularity-compose.yml -f singularity-compose.dev.yml up
+```
+
+The `singularity-compose.yml` file might specify a `webapp` instance:
+
+```yaml
+instances:
+  webapp:
+    image: webapp.sif
+    start:
+      args: "start-daemon"
+    port:
+      - "80:80"
+    volumes:
+      - /mnt/shared_drive/folder:/webapp/data
+```
+
+if the `singularity-compose.dev.yml` also specifies this same service, any matching fields override 
+the previous files.
+
+```yaml
+instances:
+  webapp:
+    start:
+      args: "start-daemon -debug"
+    volumes:
+      - /home/user/folder:/webapp/data
+```
+
+The result of the examples above would be translated in runtime to:
+
+```yaml
+instances:
+  webapp:
+    image: webapp.sif
+    start:
+      args: "start-daemon -debug"
+    port:
+      - "80:80"
+    volumes:
+      - /home/user/folder:/webapp/data
+```
+
+## project-name
+
+Specify project name.
+
+Default value: `$PWD`
+
+Aliases `--project-name`, `-p`. 
+
+```bash
+$ singularity-compose --project-name 'my_cool_project' up
+```
+
+## project-directory
+
+Specify project working directory
+
+Default value: compose file location
+
+
+```bash
+$ singularity-compose --project-directory /home/user/myfolder up
+```
+
 [home](/README.md#singularity-compose)

docs/spec/spec-2.0.md

@@ -293,8 +293,7 @@ The fields for instances are discussed below:
 
 |Name| Description |
 |----|-------------|
-|name|The name of the instance will be "nginx" unless the user provides a "name"
-field (not defined above).|
+|name|The name of the instance will be "nginx" unless the user provides a "name" field (not defined above).|
 |build| a section to define how and where to build the base container from.|
 |build.context| the folder with the Singularity file (and other relevant files). Must exist.
 |build.recipe| the Singularity recipe in the build context folder. It defaults to `Singularity`|

scompose/client/__init__.py

@@ -53,8 +53,9 @@ def get_parser():
         "--file",
         "-f",
         dest="file",
-        help="specify compose file (default singularity-compose.yml)",
-        default="singularity-compose.yml",
+        help="specify compose file (default singularity-compose.yml). It can be used multiple times",
+        action="append",
+        default=[],
     )
 
     parser.add_argument(
@@ -96,6 +97,20 @@ def get_parser():
 
     build = subparsers.add_parser("build", help="Build or rebuild containers")
 
+    # Check
+
+    check = subparsers.add_parser(
+        "check", help="check or validate singularity-compose.yml"
+    )
+
+    check.add_argument(
+        "--preview",
+        dest="preview",
+        help="print compose file(s) interpolated content",
+        default=False,
+        action="store_true",
+    )
+
     # Config
 
     config = subparsers.add_parser("config", help="Validate and view the compose file")
@@ -150,7 +165,7 @@ def get_parser():
 
     execute = subparsers.add_parser("exec", help="execute a command to an instance")
 
-    run = subparsers.add_parser("run", help="run an instance runscript.")
+    run = subparsers.add_parser("run", help="run an instance runscript")
 
     shell = subparsers.add_parser("shell", help="shell into an instance")
 
@@ -225,9 +240,16 @@ def show_help(return_code=0):
         print(scompose.__version__)
         sys.exit(0)
 
+    # argparse inherits a funny behaviour that appends default values to the dest value whether you've specified a value
+    # or not. The bug/behaviour is documented here: https://bugs.python.org/issue16399
+    if len(args.file) == 0:
+        args.file = ["singularity-compose.yml"]
+
     # Does the user want a shell?
     if args.command == "build":
         from scompose.client.build import main
+    elif args.command == "check":
+        from scompose.client.check import main
     elif args.command == "create":
         from scompose.client.create import main
     elif args.command == "config":

scompose/client/check.py

@@ -0,0 +1,37 @@
+"""
+
+Copyright (C) 2021 Vanessa Sochat.
+
+This Source Code Form is subject to the terms of the
+Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed
+with this file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+"""
+
+from scompose.logger import bot
+from scompose.config import merge_config
+from scompose.config.validate import validate_config
+import yaml
+
+
+def main(args, parser, extra):
+    """
+    Validate compose files for correctness.
+
+    CLI Arguments
+    ==========
+        --preview flag to show combined configs.
+    """
+
+    # validate compose files
+    for f in args.file:
+        result = validate_config(f)
+        if not result and not args.preview:
+            bot.info("%s is valid." % f)
+        elif result:
+            bot.exit("%s is not valid." % f)
+
+    if args.preview:
+        # preview
+        config = merge_config(args.file)
+        print(yaml.dump(config, sort_keys=False))