Update docs and examples to reflect invoke changes

invoke.yml.example

@@ -6,6 +6,7 @@ nautobot:
   compose_dir: "/full/path/to/nautobot/development"
   compose_files:
     - "docker-compose.yml"
+    - "docker-compose.dev.yml"
     - "docker-compose.override.yml"
   docker_image_names_main:
     - "networktocode/nautobot"

nautobot/docs/development/getting-started.md

@@ -171,7 +171,7 @@ Available tasks:
 A development environment can be easily started up from the root of the project using the following commands:
 
 - `invoke build` - Builds Nautobot docker images
-- `invoke migrate` - Performs database migration operation in Django    
+- `invoke migrate` - Performs database migration operation in Django
 - `invoke createsuperuser` - Creates a superuser account for the Nautobot application
 - `invoke debug` - Starts Docker containers for Nautobot, PostgreSQL, Redis, Celery, and the RQ worker in debug mode and attaches their output to the terminal in the foreground. You may enter Control-C to stop the containers.
 
@@ -184,17 +184,18 @@ Additional useful commands for the development environment:
 
 The Invoke tasks have some default [configuration](http://docs.pyinvoke.org/en/stable/concepts/configuration.html) which you may want to override. Configuration properties include:
 
+- `project_name`: The name that all Docker containers will be grouped together under (default: `nautobot`, resulting in containers named `nautobot_nautobot_1`, `nautobot_redis_1`, etc.)
 - `python_ver`: the Python version which is used to build the Docker container (default: `3.7`)
 - `local`: run the commands in the local environment vs the Docker container (default: `False`)
 - `compose_dir`: the full path to the directory containing the Docker Compose YAML files (default: `"<nautobot source directory>/development"`)
-- `compose_file`: the Docker Compose YAML file to use (default: `"docker-compose.yml"`)
-- `compose_override_file`: the default Docker Compose override file to use if it exists (default: `"docker-compose.override.yml"`)
+- `compose_files`: the Docker Compose YAML file(s) to use (default: `["docker-compose.yml", "docker-compose.dev.yml"]`)
+- `docker_image_names_main` and `docker_image_names_develop`: Used when [building Docker images for publication](release-checklist.md#publish-docker-images); you shouldn't generally need to change these.
 
 These setting may be overridden several different ways (from highest to lowest precedence):
 
 - Command line argument on the individual commands (see `invoke $command --help`) if available
-- Using environment variables such as `INVOKE_NAUTOBOT_PYTHON_VER`; the variables are prefixed with `INVOKE_NAUTOBOT_` and must be uppercase
-- Using an `invoke.yml` file (see `invoke.yml.example`)
+- Using environment variables such as `INVOKE_NAUTOBOT_PYTHON_VER`; the variables are prefixed with `INVOKE_NAUTOBOT_` and must be uppercase; note that Invoke does not presently support environment variable overriding of list properties such as `compose_files`.
+- Using an `invoke.yml` file (see [`invoke.yml.example`](https://github.com/nautobot/nautobot/blob/main/invoke.yml.example))
 
 #### Working with Docker Compose
 
@@ -219,17 +220,27 @@ In the `docker` directory you will find the following files:
 
 #### Docker-Compose Overrides
 
-If you require changing any of the defaults found in `docker-compose.yml`,  create a file inside the```development``` directory called ```docker-compose.override.yml``` and set the environment variable `INVOKE_NAUTOBOT_COMPOSE_OVERRIDE_FILE=docker-compose.override.yml`.
+If you require changing any of the defaults found in `docker-compose.yml`,  create a file inside the `development` directory called `docker-compose.override.yml` and add this file to the `compose_files` setting in your `invoke.yml` file, for example:
+
+
+```yaml
+---
+nautobot:
+  compose_files:
+    - "docker-compose.yml"
+    - "docker-compose.dev.yml"
+    - "docker-compose.override.yml"
+```
 
 This file will override any configuration in the main `docker-compose.yml` file, without making changes to the repository.
 
 Please see the [official documentation on extending Docker Compose](https://docs.docker.com/compose/extends/) for more information.
 
 ##### Automatically Creating a Superuser
 
-There may be times where you want to bootstrap Nautobot with a superuser account and API token already created for quick access or for running within a CI/CD pipeline. Below will detail the steps required to bootstrap Nautobot with a user and token.
+There may be times where you want to bootstrap Nautobot with a superuser account and API token already created for quick access or for running within a CI/CD pipeline. By using a custom `invoke.yml` as described above, in combination with custom `docker-compose.override.yml` and `override.env` files, you can automatically bootstrap Nautobot with a user and token.
 
-Create `development/docker-compose.override.yml` with the following contents:
+Create `invoke.yml` as described above, then create `development/docker-compose.override.yml` with the following contents:
 
 ```yaml
 ---
@@ -252,9 +263,6 @@ NAUTOBOT_SUPERUSER_PASSWORD=admin
 NAUTOBOT_SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567
 ```
 
-!!! warning
-    Please name the **.env** file ``override.env`` to prevent credentials from accidentally being checked into Git as ``override.env`` is set in the ``.gitignore`` file.
-
 The variables defined above within `override.env` will signal the `docker-entrypoint.sh` script to create the superuser with the specified username, email, password, and API token.
 
 After these two files are created, you can use the `invoke` tasks to manage the development containers.
@@ -527,7 +535,7 @@ Throughout the course of development, it's a good idea to occasionally run Nauto
 
 Unit tests are automated tests written and run to ensure that a section of the Nautobot application (known as the "unit") meets its design and behaves as intended and expected. Most commonly as a developer of or contributor to Nautobot you will be writing unit tests to exercise the code you have written. Unit tests are not meant to test how the application behaves, only the individual blocks of code, therefore use of mock data and phony connections is common in unit test code. As a guiding principle, unit tests should be fast, because they will be executed quite often.
 
-By Nautobot convention, unit tests must be [tagged](https://docs.djangoproject.com/en/stable/topics/testing/tools/#tagging-tests) with `unit`. The base test case class `nautobot.utilities.testing.TestCase` has this tag, therefore any test cases inheriting from that class do not need to be explicitly tagged. All existing view and API test cases in the Nautobot test suite utilities inherit from this class. 
+By Nautobot convention, unit tests must be [tagged](https://docs.djangoproject.com/en/stable/topics/testing/tools/#tagging-tests) with `unit`. The base test case class `nautobot.utilities.testing.TestCase` has this tag, therefore any test cases inheriting from that class do not need to be explicitly tagged. All existing view and API test cases in the Nautobot test suite utilities inherit from this class.
 
 !!! warning
     New unit tests **must always** inherit from `nautobot.utilities.testing.TestCase`. Do not use `django.test.TestCase`.
@@ -577,7 +585,7 @@ Integration tests are automated tests written and run to ensure that the Nautobo
 
 Integration testing is much more involved, and builds on top of the foundation laid by unit testing. As a guiding principle, integration tests should be comprehensive, because they are the last mile to asserting that Nautobot does what it is advertised to do. Without integration testing, we have to do it all manually, and that's no fun for anyone!
 
-Running integrations tests requires the use of Docker at this time. They can be directly invoked using `nautobot-server test` just as unit tests can, however, a headless Firefox browser provided by Selenium is required. Because Selenium installation and setup is complicated, we have included a configuration for this to work out of the box using Docker. 
+Running integrations tests requires the use of Docker at this time. They can be directly invoked using `nautobot-server test` just as unit tests can, however, a headless Firefox browser provided by Selenium is required. Because Selenium installation and setup is complicated, we have included a configuration for this to work out of the box using Docker.
 
 The Selenium container is running a standalone, headless Firefox "web driver" browser that can be remotely controlled by Nautobot for use in integration testing.
 
@@ -587,7 +595,7 @@ Before running integration tests, the `selenium` container must be running. If y
 |---------------------------|-----------------------------------|
 | (automatic)               | `invoke start --service selenium` |
 
-By Nautobot convention, integration tests must be [tagged](https://docs.djangoproject.com/en/stable/topics/testing/tools/#tagging-tests) with `integration`. The base test case class `nautobot.utilities.testing.integration.SeleniumTestCase` has this tag, therefore any test cases inheriting from that class do not need to be explicitly tagged. All existing integration test cases in the Nautobot test suite utilities inherit from this class. 
+By Nautobot convention, integration tests must be [tagged](https://docs.djangoproject.com/en/stable/topics/testing/tools/#tagging-tests) with `integration`. The base test case class `nautobot.utilities.testing.integration.SeleniumTestCase` has this tag, therefore any test cases inheriting from that class do not need to be explicitly tagged. All existing integration test cases in the Nautobot test suite utilities inherit from this class.
 
 !!! warning
     New integration tests **must always** inherit from `nautobot.utilities.testing.integration.SeleniumTestCase` and added in the `integration` directory in the `tests` directory of an inner Nautobot application. Do not use any other base class for integration tests.
@@ -619,7 +627,7 @@ Integration tests are run using the `invoke integration-test` command. All integ
 | `invoke integration-test` | `nautobot-server --config=nautobot/core/tests/nautobot_config.py test --tag integration nautobot` |
 
 !!! info
-    The same arguments supported by `invoke unittest` are supported by `invoke integration-test`. The key difference being the dependency upon the Selenium container, and inclusion of the `integration` tag. 
+    The same arguments supported by `invoke unittest` are supported by `invoke integration-test`. The key difference being the dependency upon the Selenium container, and inclusion of the `integration` tag.
 
 !!! tip
     You may also use `invoke integration-test` in the Virtual Environment workflow given that the `selenium` container is running, and that the `INVOKE_NAUTOBOT_LOCAL=True` environment variable has been set.

nautobot/docs/development/release-checklist.md

@@ -6,7 +6,7 @@ This document is intended for Nautobot maintainers and covers the steps to perfo
 
 ### Update Requirements
 
-Required Python packages are maintained in two files: `pyproject.toml` and `poetry.lock`. 
+Required Python packages are maintained in two files: `pyproject.toml` and `poetry.lock`.
 
 #### The `pyproject.toml` file
 Python packages are defined inside of `pyproject.toml`. The `[tool.poetry.dependencies]` section of this file contains a list of all the packages required by Nautobot.
@@ -177,12 +177,24 @@ for ver in 3.6 3.7 3.8 3.9; do
 done
 ```
 
-Test the images locally:
+Test the images locally - to do this you need to set the following in your `invoke.yml`:
+
+```yaml
+---
+nautobot:
+  compose_files:
+    - "docker-compose.yml"
+    - "docker-compose.build.yml"
+```
+
+!!! warning
+    You should *not* include `docker-compose.dev.yml` in this test scenario!
+
 ```no-highlight
 for ver in 3.6 3.7 3.8 3.9; do
   export INVOKE_NAUTOBOT_PYTHON_VER=$ver
-  INVOKE_NAUTOBOT_COMPOSE_OVERRIDE_FILE=docker-compose.build.yml invoke stop
-  INVOKE_NAUTOBOT_COMPOSE_OVERRIDE_FILE=docker-compose.build.yml invoke integration-tests
+  invoke stop
+  invoke integration-tests
 done
 ```
 

nautobot/docs/docker/index.md

@@ -128,10 +128,19 @@ REPOSITORY                                                                TAG
 networktocode/nautobot-dev                                                local                  25487d93fc1f   16 seconds ago   630MB
 ```
 
-If you need to build or test the final image set the `INVOKE_NAUTOBOT_COMPOSE_OVERRIDE_FILE` environment variable:
+If you need to build or test the final image, you must set your `invoke.yml` to use `docker-compose.build.yml` in place of `docker-compose.dev.yml`:
+
+```yaml
+---
+nautobot:
+  compose_files:
+    - "docker-compose.yml"
+    - "docker-compose.build.yml"
+```
+
+Then you can re-run the `invoke build` command:
 
 ```no-highlight
-$ export INVOKE_NAUTOBOT_COMPOSE_OVERRIDE_FILE=docker-compose.build.yml
 $ invoke build
 ...
 $ docker images

tasks.py

@@ -51,15 +51,21 @@ def is_truthy(arg):
                 "docker-compose.yml",
                 "docker-compose.dev.yml",
             ],
+            # Image names to use when building from "main" branch
             "docker_image_names_main": [
+                # Production containers - not containing development tools
                 "networktocode/nautobot",
                 "ghcr.io/nautobot/nautobot",
+                # Development containers - include development tools like linters
                 "networktocode/nautobot-dev",
                 "ghcr.io/nautobot/nautobot-dev",
             ],
+            # Image names to use when building from "develop" branch
             "docker_image_names_develop": [
+                # Production containers - not containing development tools
                 "networktocode/nautobot",
                 "ghcr.io/nautobot/nautobot",
+                # Development containers - include development tools like linters
                 "networktocode/nautobot-dev",
                 "ghcr.io/nautobot/nautobot-dev",
             ],
@@ -222,8 +228,10 @@ def docker_push(context, branch, commit="", datestamp=""):
     for image_name in docker_image_names:
         for image_tag in docker_image_tags:
             if image_name.endswith("-dev"):
+                # Use the development image as the basis for this tag and push
                 local_image = f"networktocode/nautobot-dev-py{context.nautobot.python_ver}:local"
             else:
+                # Use the production image as the basis for this tag and push
                 local_image = f"networktocode/nautobot-py{context.nautobot.python_ver}:local"
             new_image = f"{image_name}:{image_tag}"
             tag_command = f"docker tag {local_image} {new_image}"