Flexible mapping provider to parser

CHANGELOG.md

@@ -1,21 +1,30 @@
 # Changelog
 
+## v0.1.8 -
+
+### Added
+
+- #51: Add a Custom Field in Provider, `provider_parser_circuit_maintenances` to allow custom mapping of the provider type class used from the `circuit-maintenance-parser` library
+
+### Changed
+
+- #51: Improve Development Environment and upgrade Nautobot version to 1.1.0
+
 ## v0.1.7 - 2021-07-27
 
 ### Added
 
 - #42:
-    - Add stack trace to job log on exception
-    - IMAP and GMail notification sources now support a `source_header` configuration parameter to allow for cases where `From` is not the relevant header to inspect.
-
+  - Add stack trace to job log on exception
+  - IMAP and GMail notification sources now support a `source_header` configuration parameter to allow for cases where `From` is not the relevant header to inspect.
 
 ### Fixed
 
 - #42:
-    - Avoid an exception if some Providers do not have a populated `emails_circuit_maintenance` value
-    - `extract_email_source()` now correctly handles email addresses containing dash characters.
-    - Avoid an exception on processing a non-multipart email payload
-    - Don't try to create a `RawNotification` if no `raw_payload` could be extracted from the notification.
+  - Avoid an exception if some Providers do not have a populated `emails_circuit_maintenance` value
+  - `extract_email_source()` now correctly handles email addresses containing dash characters.
+  - Avoid an exception on processing a non-multipart email payload
+  - Don't try to create a `RawNotification` if no `raw_payload` could be extracted from the notification.
 
 ## v0.1.6 - 2021-07-14
 

README.md

@@ -45,10 +45,12 @@ PLUGINS_CONFIG = {
 
 ### 1. Define source emails per Provider
 
-In the Nautobot UI, under **Circuits -> Providers**, for each Provider that we would like to track via the Circuit Maintenance plugin, we must configure at least one email source address (or a comma-separated list of addresses) in the **`Custom Fields -> Emails for Circuit Maintenance plugin** field.
+In the Nautobot UI, under **Circuits -> Providers**, for each Provider that we would like to track via the Circuit Maintenance plugin, we **must** configure at least one email source address (or a comma-separated list of addresses) in the **`Custom Fields -> Emails for Circuit Maintenance plugin** field.
 
 These are the source email addresses that the plugin will detect and will use to classify each notification for each specific provider.
 
+Also, by default, the Provider **slug** is used to match the provider parser from the `circuit-maintenance-parser` library, but if a custom mapping is desired (i.e. CentruryLink to Lumen), you can define this custom mapping in the **`Custom Fields -> Provider Parser for Circuit Maintenance plugin** field.
+
 ### 2. Configure Notification Sources
 
 Notification Sources are defined in two steps:
@@ -146,14 +148,13 @@ To create a [OAuth 2.0](https://developers.google.com/identity/protocols/oauth2/
 
 > Typically the `url` setting to configure in your `nautobot_config.py` for use with OAuth integration will be `"https://accounts.google.com/o/oauth2/auth"`.
 
-
 #### 2.2 Add `Providers` to the Notification Sources
 
 In the Circuit Maintenance plugin UI section, there is a **Notification Sources** button (yellow) where you can configure the Notification Sources to track new circuit maintenance notifications from specific providers.
 
 Because the Notification Sources are defined by the configuration, you can only view and edit `providers`, but not `add` or `delete` new Notification Sources via UI or API.
 
-> Note that for emails from a given Provider to be processed, you must *both* define a source email address(es) for that Provider (Usage section 1, above) *and* add that provider to a specific Notification Source as described in this section.
+> Note that for emails from a given Provider to be processed, you must _both_ define a source email address(es) for that Provider (Usage section 1, above) _and_ add that provider to a specific Notification Source as described in this section.
 
 ### 3. Run Handle Notifications Job
 

development/creds.example.env

@@ -1,5 +1,5 @@
 POSTGRES_PASSWORD=notverysecurepwd
-REDIS_PASSWORD=notverysecurepwd
+NAUTOBOT_REDIS_PASSWORD=notverysecurepwd
 SECRET_KEY=r8OwDznj!!dci#P9ghmRfdu1Ysxm0AiPeDCQhKE+N_rClfWNj
 NAUTOBOT_CREATE_SUPERUSER=true
 NAUTOBOT_SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567

development/dev.env

@@ -9,12 +9,13 @@ NAUTOBOT_ROOT=/opt/nautobot
 POSTGRES_DB=nautobot
 POSTGRES_HOST=postgres
 POSTGRES_USER=nautbot
-REDIS_HOST=redis
-REDIS_PORT=6379
+NAUTOBOT_REDIS_HOST=redis
+NAUTOBOT_REDIS_PORT=6379
 # REDIS_SSL=True
 # Uncomment REDIS_SSL if using SSL
 SUPERUSER_EMAIL=admin@example.com
 SUPERUSER_NAME=admin
+NAUTOBOT_LOG_LEVEL=DEBUG
 
 # Enable Oauth workflow for development, http
 OAUTHLIB_INSECURE_TRANSPORT="1"

development/docker-compose.base.yml

@@ -25,8 +25,29 @@ services:
     <<: *nautobot-base
   worker:
     entrypoint: "nautobot-server rqworker"
+    healthcheck:
+      interval: 5s
+      timeout: 5s
+      start_period: 5s
+      retries: 3
+      test: ["CMD", "nautobot-server", "health_check"]
     depends_on:
       - "nautobot"
     healthcheck:
       disable: true
     <<: *nautobot-base
+  celery_worker:
+    healthcheck:
+      interval: 5s
+      timeout: 5s
+      start_period: 5s
+      retries: 3
+      test: ["CMD", "nautobot-server", "health_check"]
+    depends_on:
+      - nautobot
+      - redis
+    entrypoint:
+      - "sh"
+      - "-c" # this is to evaluate the $NAUTOBOT_LOG_LEVEL from the env
+      - "nautobot-server celery worker -B -l $$NAUTOBOT_LOG_LEVEL" ## $$ because of docker-compose
+    <<: *nautobot-base

development/docker-compose.dev.yml

@@ -3,14 +3,19 @@
 # any override will need to include these volumes to use them.
 # see:  https://github.com/docker/compose/issues/3729
 ---
+x-nautobot-base: &nautobot-base
+  volumes:
+    - "./nautobot_config.py:/opt/nautobot/nautobot_config.py"
+    - "../:/source"
+
 version: "3.4"
 services:
   nautobot:
+    healthcheck:
+      disable: true
     command: "nautobot-server runserver 0.0.0.0:8080"
-    volumes:
-      - "./nautobot_config.py:/opt/nautobot/nautobot_config.py"
-      - "../:/source"
+    <<: *nautobot-base
+  celery_worker:
+    <<: *nautobot-base
   worker:
-    volumes:
-      - "./nautobot_config.py:/opt/nautobot/nautobot_config.py"
-      - "../:/source"
+    <<: *nautobot-base

development/docker-compose.requirements.yml

@@ -8,18 +8,18 @@ services:
       - "creds.env"
     volumes:
       - "postgres_data:/var/lib/postgresql/data"
-    ports:
-      - "5432:5432"
+    # ports:
+    #   - "5432:5432"
   redis:
     image: "redis:6-alpine"
     command:
       - "sh"
       - "-c" # this is to evaluate the $REDIS_PASSWORD from the env
-      - "redis-server --appendonly yes --requirepass $$REDIS_PASSWORD"
+      - "redis-server --appendonly yes --requirepass $$NAUTOBOT_REDIS_PASSWORD"
     env_file:
       - "dev.env"
       - "creds.env"
-    ports:
-      - "6379:6379"
+    # ports:
+    #   - "6379:6379"
 volumes:
   postgres_data: {}

development/nautobot_config.py

@@ -1,15 +1,12 @@
-#########################
-#                       #
-#   Required settings   #
-#                       #
-#########################
+"""Nautobot development configuration file."""
 
 import os
 import sys
 
-from distutils.util import strtobool
 from django.core.exceptions import ImproperlyConfigured
-from nautobot.core import settings
+from nautobot.core.settings import *  # noqa: F403
+from nautobot.core.settings_funcs import is_truthy, parse_redis_connection
+
 
 # Enforce required configuration parameters
 for key in [
@@ -18,29 +15,12 @@
     "POSTGRES_USER",
     "POSTGRES_HOST",
     "POSTGRES_PASSWORD",
-    "REDIS_HOST",
-    "REDIS_PASSWORD",
     "SECRET_KEY",
 ]:
     if not os.environ.get(key):
         raise ImproperlyConfigured(f"Required environment variable {key} is missing.")
 
 
-def is_truthy(arg):
-    """Convert "truthy" strings into Booleans.
-
-    Examples:
-        >>> is_truthy('yes')
-        True
-    Args:
-        arg (str): Truthy string (True values are y, yes, t, true, on and 1; false values are n, no,
-        f, false, off and 0. Raises ValueError if val is anything else.
-    """
-    if isinstance(arg, bool):
-        return arg
-    return bool(strtobool(arg))
-
-
 TESTING = len(sys.argv) > 1 and sys.argv[1] == "test"
 
 # This is a list of valid fully-qualified domain names (FQDNs) for the Nautobot server. Nautobot will not permit write
@@ -63,62 +43,22 @@ def is_truthy(arg):
     }
 }
 
-# Nautobot uses RQ for task scheduling. These are the following defaults.
-# For detailed configuration see: https://github.com/rq/django-rq#installation
-RQ_QUEUES = {
-    "default": {
-        "HOST": os.getenv("REDIS_HOST", "localhost"),
-        "PORT": os.getenv("REDIS_PORT", 6379),
-        "DB": 0,
-        "PASSWORD": os.getenv("REDIS_PASSWORD", ""),
-        "SSL": os.getenv("REDIS_SSL", False),
-        "DEFAULT_TIMEOUT": 300,
-    },
-    "webhooks": {
-        "HOST": os.getenv("REDIS_HOST", "localhost"),
-        "PORT": os.getenv("REDIS_PORT", 6379),
-        "DB": 0,
-        "PASSWORD": os.getenv("REDIS_PASSWORD", ""),
-        "SSL": os.getenv("REDIS_SSL", False),
-        "DEFAULT_TIMEOUT": 300,
-    },
-    "custom_fields": {
-        "HOST": os.getenv("REDIS_HOST", "localhost"),
-        "PORT": os.getenv("REDIS_PORT", 6379),
-        "DB": 0,
-        "PASSWORD": os.getenv("REDIS_PASSWORD", ""),
-        "SSL": os.getenv("REDIS_SSL", False),
-        "DEFAULT_TIMEOUT": 300,
-    },
-    # "with-sentinel": {
-    #     "SENTINELS": [
-    #         ("mysentinel.redis.example.com", 6379)
-    #         ("othersentinel.redis.example.com", 6379)
-    #     ],
-    #     "MASTER_NAME": 'nautobot",
-    #     "DB": 0,
-    #     "PASSWORD": "",
-    #     "SOCKET_TIMEOUT": None,
-    #     'CONNECTION_KWARGS': {
-    #         'socket_connect_timeout': 10,
-    #     },
-    # },
-    "check_releases": {
-        "HOST": os.getenv("REDIS_HOST", "localhost"),
-        "PORT": os.getenv("REDIS_PORT", 6379),
-        "DB": 0,
-        "PASSWORD": os.getenv("REDIS_PASSWORD", ""),
-        "SSL": os.getenv("REDIS_SSL", False),
-        "DEFAULT_TIMEOUT": 300,
-    },
-}
 
 # Nautobot uses Cacheops for database query caching. These are the following defaults.
 # For detailed configuration see: https://github.com/Suor/django-cacheops#setup
-REDIS_HOST = os.getenv("REDIS_HOST", "localhost")
-REDIS_PORT = os.getenv("REDIS_PORT", 6379)
-REDIS_PASS = os.getenv("REDIS_PASSWORD", "")
-CACHEOPS_REDIS = f"redis://:{REDIS_PASS}@{REDIS_HOST}:{REDIS_PORT}/1"
+
+CACHES = {
+    "default": {
+        "BACKEND": "django_redis.cache.RedisCache",
+        "LOCATION": parse_redis_connection(redis_database=0),
+        "TIMEOUT": 300,
+        "OPTIONS": {
+            "CLIENT_CLASS": "django_redis.client.DefaultClient",
+        },
+    }
+}
+CACHEOPS_REDIS = parse_redis_connection(redis_database=1)
+
 
 # This key is used for secure generation of random numbers and strings. It must never be exposed outside of this file.
 # For optimal security, SECRET_KEY should be at least 50 characters in length and contain a mix of letters, numbers, and
@@ -383,7 +323,7 @@ def is_truthy(arg):
 # Django Debug Toolbar
 DEBUG_TOOLBAR_CONFIG = {"SHOW_TOOLBAR_CALLBACK": lambda _request: DEBUG and not TESTING}
 
-if "debug_toolbar" not in EXTRA_INSTALLED_APPS:
-    EXTRA_INSTALLED_APPS.append("debug_toolbar")
-if "debug_toolbar.middleware.DebugToolbarMiddleware" not in settings.MIDDLEWARE:
-    settings.MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")
+if "debug_toolbar" not in INSTALLED_APPS:  # noqa: F405
+    INSTALLED_APPS.append("debug_toolbar")  # noqa: F405
+if "debug_toolbar.middleware.DebugToolbarMiddleware" not in MIDDLEWARE:  # noqa: F405
+    MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")  # noqa: F405

nautobot_circuit_maintenance/__init__.py

@@ -6,8 +6,8 @@
 from nautobot.extras.plugins import PluginConfig
 
 
-def custom_field_extension(sender, **kwargs):  # pylint: disable=unused-argument
-    """Add extended custom field."""
+def custom_fields_extension(sender, **kwargs):  # pylint: disable=unused-argument
+    """Add extended custom fields."""
     # pylint: disable=import-outside-toplevel
     from django.contrib.contenttypes.models import ContentType
     from nautobot.circuits.models import Provider
@@ -18,7 +18,12 @@ def custom_field_extension(sender, **kwargs):  # pylint: disable=unused-argument
         {
             "name": "emails_circuit_maintenances",
             "type": CustomFieldTypeChoices.TYPE_TEXT,
-            "label": "Emails for Circuit Maintenance plugin",
+            "label": "Emails for Circuit Maintenance plugin.",
+        },
+        {
+            "name": "provider_parser_circuit_maintenances",
+            "type": CustomFieldTypeChoices.TYPE_TEXT,
+            "label": "Provider Parser for Circuit Maintenance plugin.",
         },
     ]:
         field, _ = CustomField.objects.get_or_create(name=provider_cf_dict["name"], defaults=provider_cf_dict)
@@ -66,7 +71,7 @@ class CircuitMaintenanceConfig(PluginConfig):
 
     def ready(self):
         super().ready()
-        post_migrate.connect(custom_field_extension, sender=self)
+        post_migrate.connect(custom_fields_extension, sender=self)
         post_migrate.connect(import_notification_sources, sender=self)
 
 

nautobot_circuit_maintenance/custom_validators.py

@@ -1,6 +1,9 @@
 """Custom Validators definition."""
 from nautobot.circuits.models import Provider
 from nautobot.extras.plugins import PluginCustomValidator
+from nautobot.extras.models import CustomField
+
+from circuit_maintenance_parser import SUPPORTED_PROVIDER_NAMES
 
 
 class ProviderEmailValidator(PluginCustomValidator):
@@ -26,4 +29,28 @@ def clean(self):
                         )
 
 
-custom_validators = [ProviderEmailValidator]
+class ProviderParserValidator(PluginCustomValidator):
+    """Custom validator to validate that Provider's parser exists in the Parser library."""
+
+    model = "circuits.provider"
+
+    def clean(self):
+        """Validate that the Provider's parser exists in the Parser library."""
+        provider_mapping = (
+            self.context["object"]
+            .get_custom_fields()
+            .get(CustomField.objects.get(name="provider_parser_circuit_maintenances"))
+        )
+
+        if provider_mapping and provider_mapping.lower() not in SUPPORTED_PROVIDER_NAMES:
+            self.validation_error(
+                {
+                    "cf_provider_parser_circuit_maintenances": (
+                        f"{provider_mapping} is not one of the supported Providers in the "
+                        f"circuit-maintenance-parser library: {SUPPORTED_PROVIDER_NAMES}."
+                    )
+                }
+            )
+
+
+custom_validators = [ProviderEmailValidator, ProviderParserValidator]