Merge pull request #5136 from nautobot/release/2.1.2

* Added global filtering to Job Result log table * Fix integration test failure * Apply suggestions from code review Co-authored-by: Gary Snider <75227981+gsnider2195@users.noreply.github.com> * use self.browser.is_text_not_present instead of time.sleep * PR feedback * fix migration * fixes early return conditional in ensure_git_repository - again. (#5043) The type returned by `git.Repo.rev_parse("HEAD")` was still off, i.e. not a string. This commit fixes this by converting it to a string. Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Improve IPAddressEditView and IPAddressAssignView logic (#5054) * Improve IPAddressEditView and IPAddressAssignView logic. * Number change fragments * Make ruff happy --------- Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> * bump version * Process CSV import one row at a time to permit back-references (#4977) * Process CSV import one row at a time to permit back-references to earlier rows * Fix failing test * Change fragment --------- Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> * fix: devcontainer (#4906) * fix: devcontainer * refactor(devcontainer): remove pylint and pre-commit leftover * fix(devcontainer): arch detection * feat(devcontainer): replace black, flake8 and isort with ruff * refactor(devcontainer): remove prettier and eslint extensions * refactor: remove eslint fixall * refactor(devcontainer): move superuser information to dev.env * fix(devcontainer): update comment with ruff instead of black * update docs and add changelogs * Apply suggestions from code review * move changelogs to housekeeping category --------- Co-authored-by: Gary Snider <75227981+gsnider2195@users.noreply.github.com> Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * tt * ruff fix * Fixed bug with invoke cli and invoke nbshell. (#5079) Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Increase grid breakpoints by 240px (#5080) * Update nautobot/extras/tests/integration/test_jobs.py * bump packaging (#5077) * bump packaging * update lock file * create change fragment * update packaging dependency spec to '>=23.1' * Update changes/5076.housekeeping Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> --------- Co-authored-by: Anthony House <anthony.house@networktocode.com> Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Job-related bug fixes (#5095) * Job-related bug fixes * Renumber change fragment * [Snyk] Security upgrade jinja2 from 3.1.2 to 3.1.3 (#5086) * Bump gitpython from 3.1.40 to 3.1.41 (#5083) * A few performance updates (#5024) Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> Co-authored-by: Gary Snider <75227981+gsnider2195@users.noreply.github.com> * Skip test * Removes startplugin mgmt cmd. (#5082) Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Remove `files/get` endpoint which is not used from Nautobot 2.0 (#5115) * remove `files/get` endpoint * Update nautobot/core/tests/test_views.py Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * update based on review --------- Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * more flexible default sanitizer pattern (#4994) * more flexible default sanitizer pattern * Fix re.compile syntax * add test for new patterns create changelog fragment * fix test cases for new pattern * update other santization references * add examples of matching patterns --------- Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * More job fixes (#5102) * More job fixes * Renumber change fragment * Revert redundant logging * Enhance sanitize() to handle data other than strings, fix tests * Add checks to runjob command * Ruff * Additional change fragments * Address review feedback, add test coverage * BugFix cf table entry rendering (#5081) * BugFix cf table entry rendering * Update nautobot/core/tables.py Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Create 5081.fixed --------- Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Fix #4075 - sortability of Device Bays list view by installed device status (#5110) Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> * Update PULL_REQUEST_TEMPLATE.md (#5118) * Update PULL_REQUEST_TEMPLATE.md * Create 5118.housekeeping --------- Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> * Fixed Sync Git Repository requires non-matching permissions for UI vs API (#5128) * Adds location & rack group to device bulk edit query params (#5113) * Sanitize `render_markdown()` output with `nh3` library (#5133) * Fix GHSA-v4xv-795h-rv4h. * Renumber change fragments * Address review feedback * Test fix * Ruff * Review feedback * Update nautobot/core/forms/fields.py * Towncrier and version bump * Fixes #5058: use location instead of location_id in vm forms (#5124) * Bug: VM Interface vlan list in UI uses invalid option Fixes #5058 * added changelog fragment * grammer * Update changes/5058.fixed Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> --------- Co-authored-by: TLCF <31373129+TLCF@users.noreply.github.com> Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com> Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> * Update release note * Add `TreeManager.max_depth` cachedproperty as an alternative to `max_tree_depth()` (#5131) * Add TreeManager.max_depth cachedproperty as an alternative to max_tree_depth() * Change fragment * Add signal to clear max_depth cache when needed and test case to verify it * Update release-note --------- Co-authored-by: Timizuo <ebideritimizuo@gmail.com> Co-authored-by: Timizuo <94907097+timizuoebideri1@users.noreply.github.com> Co-authored-by: Gary Snider <75227981+gsnider2195@users.noreply.github.com> Co-authored-by: Leo Kirchner <Kircheneer@users.noreply.github.com> Co-authored-by: Hanlin Miao <46973263+HanlinMiao@users.noreply.github.com> Co-authored-by: Eric Jacob <erjac77@gmail.com> Co-authored-by: housepbass <80693460+housepbass@users.noreply.github.com> Co-authored-by: Anthony House <anthony.house@networktocode.com> Co-authored-by: Bryan Culver <31187+bryanculver@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: John Anderson <lampwins@gmail.com> Co-authored-by: Josh VanDeraa <jv@networktocode.com> Co-authored-by: Gerasimos Tzakis <gertzakis@gmail.com> Co-authored-by: Jeff Kala <48843785+jeffkala@users.noreply.github.com> Co-authored-by: Jacob McGill <9847006+jmcgill298@users.noreply.github.com> Co-authored-by: Joe Wesch <10467633+joewesch@users.noreply.github.com> Co-authored-by: TL <9435779+tlourey@users.noreply.github.com> Co-authored-by: TLCF <31373129+TLCF@users.noreply.github.com>
nautobot · Jan 22, 2024 · 4b7fa19 · 4b7fa19
2 parents cf31f79 + 0ca4d7b
commit 4b7fa19
Show file tree

Hide file tree

Showing 97 changed files with 1,252 additions and 621 deletions.
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -1,19 +1,62 @@
 {
-    "dockerComposeFile": ["../development/docker-compose.yml", "../development/docker-compose.debug.yml"],
+    "name": "Nautobot Dev Container",
+    "dockerComposeFile": [
+        "../development/docker-compose.yml",
+        "../development/docker-compose.postgres.yml",
+        "../development/docker-compose.dev.yml"
+    ],
+    "features": {
+        "ghcr.io/devcontainers/features/docker-outside-of-docker:1": { }
+    },
     "service": "nautobot",
-	"workspaceFolder": "/source/",
-	"extensions": [
-		"ms-python.vscode-pylance",
-		"streetsidesoftware.code-spell-checker",
-		"eamodio.gitlens",
-		"oderwat.indent-rainbow",
-		"ms-python.python",
-		"ms-vsliveshare.vsliveshare"
-	],
-	"settings": {
-		"python.pythonPath": "/usr/local/bin/python",
-		"python.analysis.extraPaths": ["/source"],
-		"python.linting.pylintEnabled": true,
-		"python.linting.enabled": true
-	}
+    "shutdownAction": "stopCompose",
+    "customizations": {
+        "vscode": {
+            "extensions": [
+                "charliermarsh.ruff",
+                "DavidAnson.vscode-markdownlint",
+                "eamodio.gitlens",
+                "EditorConfig.EditorConfig",
+                "GitHub.vscode-pull-request-github",
+                "ms-python.python",
+                "ms-python.vscode-pylance",
+                "samuelcolvin.jinjahtml",
+                "tamasfe.even-better-toml"
+            ],
+            "settings": {
+                // Global editor settings
+                "editor.codeActionsOnSave": {
+                    "source.fixAll.markdownlint": "explicit"
+                },
+                "editor.formatOnPaste": true,
+                "editor.formatOnSave": true,
+                // JSON settings
+                "json.format.keepLines": true,
+                "workbench.settings.useSplitJSON": false,
+                // Markdown settings
+                "[markdown]": {
+                    "editor.defaultFormatter": "DavidAnson.vscode-markdownlint"
+                },
+                // Python settings
+                "[python]": {
+                    "editor.codeActionsOnSave": {
+                        "source.fixAll": "explicit",
+                        "source.organizeImports": "explicit"
+                    },
+                    // When rebuilding the devcontainer, `Developer: Reload Window` must be
+                    // executed for the Ruff formatter to be recognized by the settings.
+                    // See: https://github.com/microsoft/vscode/issues/189839
+                    "editor.defaultFormatter": "charliermarsh.ruff",
+                    "editor.formatOnSave": true
+                },
+                "python.analysis.extraPaths": [
+                    "${workspaceFolder}"
+                ],
+                "python.defaultInterpreterPath": "/usr/local/bin/python",
+                "python.pythonPath": "/usr/local/bin/python"
+            }
+        }
+    },
+    "workspaceFolder": "/source",
+    "postCreateCommand": "bash .devcontainer/postCreateCommand.sh"
 }
diff --git a/.devcontainer/postCreateCommand.sh b/.devcontainer/postCreateCommand.sh
@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+set -ex
+
+git config --global --add safe.directory /source
diff --git a/.editorconfig b/.editorconfig
@@ -0,0 +1,15 @@
+# EditorConfig is awesome: https://EditorConfig.org
+
+root = true
+
+# All
+[*]
+charset = utf-8
+end_of_line = lf
+indent_size = 4
+indent_style = space
+insert_final_newline = true
+trim_trailing_whitespace = true
+
+[*.{yaml,yml}]
+indent_size = 2
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -7,18 +7,19 @@
 
     Please indicate the relevant feature request or bug report below.
 -->
-# Closes: #<ISSUE NUMBER GOES HERE>
+# Closes #<ISSUE NUMBER GOES HERE>
 # What's Changed
 <!--
     Please include:
     - A summary of the proposed changes
     - A sectioned breakdown for larger features under ## subheadings
+-->
+# Screenshots
+<!--
     - Screenshots, example payloads where relevant:
       - Before/After for bugfixes
       - Using a new feature
 -->
-
-
 # TODO
 <!--
     Please feel free to update todos to keep track of your own notes for WIP PRs.

diff --git a/development/.env.example b/development/.env.example
@@ -0,0 +1,2 @@
+# Required for VS Code Dev Container support
+PYTHON_VER=3.11
diff --git a/development/dev.env b/development/dev.env
@@ -41,3 +41,10 @@ NAUTOBOT_SELENIUM_HOST=nautobot
 # Set max connection age to the default for Django (close at end of each request)
 # NodeJS container will cause connection exhaustion
 NAUTOBOT_DB_TIMEOUT=0
+
+# Superuser information. NAUTOBOT_CREATE_SUPERUSER defaults to false.
+NAUTOBOT_CREATE_SUPERUSER=true
+NAUTOBOT_SUPERUSER_NAME=admin
+NAUTOBOT_SUPERUSER_EMAIL=admin@example.com
+NAUTOBOT_SUPERUSER_PASSWORD=admin
+NAUTOBOT_SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567
diff --git a/development/docker-compose.yml b/development/docker-compose.yml
@@ -28,7 +28,7 @@ services:
     healthcheck:
       interval: 5s
       timeout: 5s
-      start_period: 1m
+      start_period: 5m  # it takes a WHILE to run initial migrations with an empty DB
       retries: 3
       test:
         - "CMD"

diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -1,5 +1,7 @@
 # syntax=docker/dockerfile:1
 ARG PYTHON_VER
+# Small hack to make the devcontainer work until https://github.com/devcontainers/cli/issues/275 is fixed.
+ARG ARCH=${TARGETARCH:-amd64}
 
 ################################ Overview
 
@@ -126,7 +128,7 @@ RUN poetry config virtualenvs.create ${POETRY_VIRTUALENVS_CREATE} && \
 ################################ Stage: python-dependencies (intermediate build target)
 
 # hadolint ignore=DL3006
-FROM system-dev-dependencies-$TARGETARCH AS python-dependencies
+FROM system-dev-dependencies-${ARCH} AS python-dependencies
 
 ARG POETRY_HOME=/opt/poetry
 COPY --from=poetry ${POETRY_HOME} ${POETRY_HOME}

diff --git a/examples/example_plugin/docs/requirements.txt b/examples/example_plugin/docs/requirements.txt
@@ -1,4 +1,4 @@
-Jinja2==3.1.2
+Jinja2==3.1.3
 mkdocs==1.5.3
 mkdocs-include-markdown-plugin==4.0.4
 mkdocs-material==9.1.18

diff --git a/nautobot/core/celery/backends.py b/nautobot/core/celery/backends.py
@@ -75,26 +75,6 @@ def prepare_exception(self, exc, serializer=None):
         exc_info = super().prepare_exception(exc, serializer=serializer)
 
         exc_message = exc_info["exc_message"]
-
-        # If the message is iterable, walk through every item and try to sanitize any strings.
-        if isinstance(exc_message, (list, tuple)):
-            new_exc_message = []
-            for item in exc_message:
-                if isinstance(item, list):
-                    new_list = []
-                    for i in item:
-                        if isinstance(i, str):
-                            i = sanitize(i)
-                        new_list.append(i)
-                    new_exc_message.append(new_list)
-                elif isinstance(item, bytes):
-                    new_exc_message.append(sanitize(item.decode("utf-8")))
-                elif isinstance(item, str):
-                    new_exc_message.append(sanitize(item))
-                # Pass through anything that isn't a string/list of strings
-                else:
-                    new_exc_message.append(item)
-
-            exc_info["exc_message"] = tuple(new_exc_message)
+        exc_info["exc_message"] = sanitize(exc_message)
 
         return exc_info
diff --git a/nautobot/core/constants.py b/nautobot/core/constants.py
@@ -1,3 +1,7 @@
+from copy import deepcopy
+
+import nh3
+
 SEARCH_MAX_RESULTS = 15
 
 #
@@ -32,6 +36,35 @@
 
 FILTER_NEGATION_LOOKUP_MAP = {"n": "exact"}
 
+#
+# User input sanitization
+#
+
+# Subset of the HTML tags allowed by default by ammonia:
+# https://github.com/rust-ammonia/ammonia/blob/master/src/lib.rs
+HTML_ALLOWED_TAGS = nh3.ALLOWED_TAGS - {
+    # no image maps at present
+    "area",
+    "map",
+    # no document-level markup at present
+    "article",
+    "aside",
+    "footer",
+    "header",
+    "nav",
+    # miscellaneous out-of-scope for now
+    "data",
+    "dfn",
+    "figcaption",
+    "figure",
+}
+
+# Variant of the HTML attributes allowed by default by ammonia:
+# https://github.com/rust-ammonia/ammonia/blob/master/src/lib.rs
+# at present we just copy nh3.ALLOWED_ATTRIBUTES but we can modify this later as desired and appropriate
+HTML_ALLOWED_ATTRIBUTES = deepcopy(nh3.ALLOWED_ATTRIBUTES)
+
+
 #
 # Reserved Names
 #

diff --git a/nautobot/core/forms/fields.py b/nautobot/core/forms/fields.py
@@ -9,7 +9,9 @@
 from django.core.exceptions import MultipleObjectsReturned, ObjectDoesNotExist, ValidationError
 from django.db.models import Q
 from django.forms.fields import BoundField, InvalidJSONInput, JSONField as _JSONField
+from django.templatetags.static import static
 from django.urls import reverse
+from django.utils.html import format_html
 import django_filters
 from netaddr import EUI
 from netaddr.core import AddrFormatError
@@ -372,12 +374,16 @@ class CommentField(django_forms.CharField):
 
     widget = django_forms.Textarea
     default_label = ""
-    # TODO: Port Markdown cheat sheet to internal documentation
-    default_helptext = (
-        '<i class="mdi mdi-information-outline"></i> '
-        '<a href="https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet" target="_blank">'
-        "Markdown</a> syntax is supported"
-    )
+
+    @property
+    def default_helptext(self):
+        # TODO: Port Markdown cheat sheet to internal documentation
+        return format_html(
+            '<i class="mdi mdi-information-outline"></i> '
+            '<a href="https://www.markdownguide.org/cheat-sheet/#basic-syntax" rel="noopener noreferrer">Markdown</a> '
+            'syntax is supported, as well as <a href="{}#render_markdown">a limited subset of HTML</a>.',
+            static("docs/user-guide/platform-functionality/template-filters.html"),
+        )
 
     def __init__(self, *args, **kwargs):
         required = kwargs.pop("required", False)

diff --git a/nautobot/core/management/commands/startplugin.py b/nautobot/core/management/commands/startplugin.py
diff --git a/nautobot/core/models/querysets.py b/nautobot/core/models/querysets.py
@@ -6,17 +6,23 @@
 from nautobot.core.utils.data import merge_dicts_without_collision
 
 
-def count_related(model, field):
+def count_related(model, field, *, filter_dict=None):
     """
     Return a Subquery suitable for annotating a child object count.
+
+    Args:
+        model (Model): The related model to aggregate
+        field (str): The field on the related model which points back to the OuterRef model
+        filter_dict (dict): Optional dict of filter key/value pairs to limit the Subquery
     """
+    filters = {field: OuterRef("pk")}
+    if filter_dict:
+        filters.update(filter_dict)
 
     manager = model.objects
     if hasattr(model.objects, "without_tree_fields"):
         manager = manager.without_tree_fields()
-    subquery = Subquery(
-        manager.filter(**{field: OuterRef("pk")}).order_by().values(field).annotate(c=Count("*")).values("c")
-    )
+    subquery = Subquery(manager.filter(**filters).order_by().values(field).annotate(c=Count("*")).values("c"))
 
     return Coalesce(subquery, 0)
 

diff --git a/nautobot/core/models/tree_queries.py b/nautobot/core/models/tree_queries.py
@@ -1,3 +1,5 @@
+from functools import cached_property
+
 from django.core.cache import cache
 from tree_queries.models import TreeNode
 from tree_queries.query import TreeManager as TreeManager_, TreeQuerySet as TreeQuerySet_
@@ -28,6 +30,10 @@ class TreeManager(TreeManager_, BaseManager.from_queryset(TreeQuerySet)):
     _with_tree_fields = True
     use_in_migrations = True
 
+    @cached_property
+    def max_depth(self):
+        return self.max_tree_depth()
+
 
 class TreeModel(TreeNode):
     """

diff --git a/nautobot/core/settings.py b/nautobot/core/settings.py
@@ -124,7 +124,10 @@
 SANITIZER_PATTERNS = [
     # General removal of username-like and password-like tokens
     (re.compile(r"(https?://)?\S+\s*@", re.IGNORECASE), r"\1{replacement}@"),
-    (re.compile(r"(username|password|passwd|pwd)((?:\s+is.?|:)?\s+)\S+", re.IGNORECASE), r"\1\2{replacement}"),
+    (
+        re.compile(r"(username|password|passwd|pwd|secret|secrets)([\"']?(?:\s+is.?|:)?\s+)\S+[\"']?", re.IGNORECASE),
+        r"\1\2{replacement}",
+    ),
 ]
 
 # Storage
@@ -679,7 +682,8 @@
     ),
     "SUPPORT_MESSAGE": ConstanceConfigItem(
         default="",
-        help_text="Help message to include on 4xx and 5xx error pages. Markdown is supported.\n"
+        help_text="Help message to include on 4xx and 5xx error pages. "
+        "Markdown is supported, as are some HTML tags and attributes.\n"
         "If unspecified, instructions to join Network to Code's Slack community will be provided.",
     ),
 }