Merge pull request #669 from nautobot/develop

Co-authored-by: Stephen Kiely <stephenkiely87@gmail.com>
Co-authored-by: Jeff Kala <48843785+jeffkala@users.noreply.github.com>
Co-authored-by: Glenn Matthews <glenn.matthews@networktocode.com>
Co-authored-by: tim-fiola <timothy.fiola@gmail.com>
Co-authored-by: jifox <jifox@users.noreply.github.com>
Co-authored-by: nniehoff <github@nickniehoff.net>
Co-authored-by: Ken Celenza <ken@celenza.org>
Co-authored-by: Kristian Carbonaro <carbonarok@gmail.com>
Co-authored-by: Jordan Facibene <jordan@networktocode.com>
Co-authored-by: Sammy Chafiqui <sammy.chafiqui@gmail.com>

nautobot/core/apps/__init__.py

@@ -238,10 +238,11 @@ def __init__(self, link, name, permissions=None, buttons=None, weight=1000):
         self.name = name
         self.weight = weight
 
-        if buttons is not None and not isinstance(buttons, (list, tuple)):
-            raise TypeError("Buttons must be passed as a tuple or list.")
-        elif not all(isinstance(button, NavMenuButton) for button in buttons):
-            raise TypeError("All buttons defined in an item must be an instance or subclass of NavMenuButton")
+        if buttons is not None:
+            if not isinstance(buttons, (list, tuple)):
+                raise TypeError("Buttons must be passed as a tuple or list.")
+            elif not all(isinstance(button, NavMenuButton) for button in buttons):
+                raise TypeError("All buttons defined in an item must be an instance or subclass of NavMenuButton")
         self.buttons = buttons
 
 

nautobot/core/settings.py

@@ -54,7 +54,7 @@
 NAUTOBOT_ROOT = os.getenv("NAUTOBOT_ROOT", os.path.expanduser("~/.nautobot"))
 
 CHANGELOG_RETENTION = 90
-DOCS_ROOT = os.path.join(os.path.dirname(BASE_DIR), "docs")
+DOCS_ROOT = os.path.join(BASE_DIR, "docs")
 HIDE_RESTRICTED_UI = False
 
 # By default, Nautobot will permit users to create duplicate prefixes and IP addresses in the global

nautobot/core/templates/nautobot_config.py.j2

@@ -16,7 +16,7 @@ from nautobot.core.settings_funcs import is_truthy, parse_redis_connection
 # Example: ALLOWED_HOSTS = ['nautobot.example.com', 'nautobot.internal.local']
 ALLOWED_HOSTS = os.getenv("NAUTOBOT_ALLOWED_HOSTS", "").split(" ")
 
-# PostgreSQL database configuration. See the Django documentation for a complete list of available parameters:
+# Database configuration. See the Django documentation for a complete list of available parameters:
 #   https://docs.djangoproject.com/en/stable/ref/settings/#databases
 DATABASES = {
     "default": {

nautobot/dcim/api/serializers.py

@@ -759,11 +759,13 @@ class Meta(DeviceSerializer.Meta):
             "local_context_schema",
             "local_context_data",
             "tags",
+            "computed_fields",
             "custom_fields",
             "config_context",
             "created",
             "last_updated",
         ]
+        opt_in_fields = ["computed_fields"]
 
     @swagger_serializer_method(serializer_or_field=serializers.DictField)
     def get_config_context(self, obj):

nautobot/docs/additional-features/jobs.md

@@ -16,7 +16,7 @@ Jobs are a way for users to execute custom logic on demand from within the Nauto
 
 ## Writing Jobs
 
-Jobs may be manually installed as files in the [`JOBS_ROOT`](../../configuration/optional-settings/#jobs_root) path (which defaults to `~/.nautobot/jobs/`). Each file created within this path is considered a separate module. Each module holds one or more Jobs (Python classes), each of which serves a specific purpose. The logic of each job can be split into a number of distinct methods, each of which performs a discrete portion of the overall job logic.
+Jobs may be manually installed as files in the [`JOBS_ROOT`](../../configuration/optional-settings/#jobs_root) path (which defaults to `$NAUTOBOT_ROOT/jobs/`). Each file created within this path is considered a separate module. Each module holds one or more Jobs (Python classes), each of which serves a specific purpose. The logic of each job can be split into a number of distinct methods, each of which performs a discrete portion of the overall job logic.
 
 !!! warning
     The jobs path must include a file named `__init__.py`, which registers the path as a Python module. Do not delete this file.
@@ -53,6 +53,9 @@ You can implement the entire job within the `run()` function, but for more compl
 
 It's important to understand that jobs execute on the server asynchronously as background tasks; they log messages and report their status to the database as [`JobResult`](../models/extras/jobresult.md) records.
 
+!!! note
+    When actively developing a Job utilizing a development environment it's important to understand that the "reload on code changes" debugging functionality does **not** automatically restart the `nautobot_worker`; therefore, it is required to restart the `worker` after each update to your Job source code.
+
 ### Module Attributes
 
 #### `name`

nautobot/docs/administration/nautobot-server.md

@@ -9,6 +9,9 @@ For those familiar with Django applications, this CLI utility works exactly as a
 !!! important
     Since Nautobot is a Django application, there are a number of built-in management commands that will not be covered in this document. Please see the [official Django documentation on management commands](https://docs.djangoproject.com/en/stable/ref/django-admin/#available-commands) for more information.
 
+!!! important
+    Django does not recognize `nautobot-server`.  Anywhere `python manage.py` is mentioned, it is safe to replace with `nautobot-server`.
+
 ## Getting Help
 
 To see all available management commands:
@@ -124,6 +127,25 @@ nautobot=> \q
 !!! note
     This is a built-in Django command. Please see the [official documentation on `dbshell`](https://docs.djangoproject.com/en/stable/ref/django-admin/#dbshell) for more information.
 
+
+### `fix_custom_fields`
+
+`nautobot-server fix_custom_fields`
+
+Adds/Removes any custom fields which should or should not exist on an object.  This command should not be run unless a custom fields jobs has failed:
+
+```no-highlight
+$ nautobot-server fix_custom_fields
+Processing ContentType dcim | device
+Processing ContentType dcim | site
+Processing ContentType dcim | rack
+Processing ContentType dcim | cable
+Processing ContentType dcim | power feed
+Processing ContentType circuits | circuit
+Processing ContentType ipam | prefix
+... (truncated for brevity of documentation) ...
+```
+
 ### `generate_secret_key`
 
 `nautobot-server generate_secret_key`

nautobot/docs/development/application-registry.md

@@ -4,7 +4,7 @@ The registry is an in-memory data structure which houses various application-wid
 
 The registry behaves essentially like a Python dictionary, with the notable exception that once a store (key) has been declared, it cannot be deleted or overwritten. The value of a store can, however, be modified; e.g. by appending a value to a list. Store values generally do not change once the application has been initialized.
 
-The registry can be inspected by importing `registry` from `extras.registry`.
+The registry can be inspected by importing `registry` from `nautobot.extras.registry`.
 
 ## Stores
 

nautobot/docs/development/getting-started.md

@@ -171,6 +171,7 @@ Available tasks:
 A development environment can be easily started up from the root of the project using the following commands:
 
 - `invoke build` - Builds Nautobot docker images
+- `invoke migrate` - Performs database migration operation in Django    
 - `invoke createsuperuser` - Creates a superuser account for the Nautobot application
 - `invoke debug` - Starts Docker containers for Nautobot, PostgreSQL, Redis, Celery, and the RQ worker in debug mode and attaches their output to the terminal in the foreground. You may enter Control-C to stop the containers.
 
@@ -243,12 +244,12 @@ The `docker-entrypoint.sh` script will run any migrations and then look for spec
  Any variables defined in this file will override the defaults. The `override.env` should look like the following:
 
 ```bash
-# Superuser information. CREATE_SUPERUSER defaults to false.
-CREATE_SUPERUSER=true
-SUPERUSER_NAME=admin
-SUPERUSER_EMAIL=admin@example.com
-SUPERUSER_PASSWORD=admin
-SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567
+# Superuser information. NAUTOBOT_CREATE_SUPERUSER defaults to false.
+NAUTOBOT_CREATE_SUPERUSER=true
+NAUTOBOT_SUPERUSER_NAME=admin
+NAUTOBOT_SUPERUSER_EMAIL=admin@example.com
+NAUTOBOT_SUPERUSER_PASSWORD=admin
+NAUTOBOT_SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567
 ```
 
 !!! warning

nautobot/docs/installation/services.md

@@ -76,14 +76,23 @@ log-5xx = true
 ; processes = 5
 
 ; If using subdirectory hosting e.g. example.com/nautobot, you must uncomment this line. Otherwise you'll get double paths e.g. example.com/nautobot/nautobot/.
-; See: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.11.html#fixpathinfo-routing-action
+; Ref: https://uwsgi-docs.readthedocs.io/en/latest/Changelog-2.0.11.html#fixpathinfo-routing-action
 ; route-run = fixpathinfo:
+
+; If hosted behind a load balancer uncomment these lines, the harakiri timeout should be greater than your load balancer timeout.
+; Ref: https://uwsgi-docs.readthedocs.io/en/latest/HTTP.html?highlight=keepalive#http-keep-alive
+; harakiri = 65
+; add-header = Connection: Keep-Alive
+; http-keepalive = 1
 ```
 
 This configuration should suffice for most initial installations, you may wish to edit this file to change the bound IP
 address and/or port number, or to make performance-related adjustments. See [uWSGI
 documentation](https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html) for the available configuration parameters.
 
+!!! note
+    If you are deploying uWSGI behind a load balancer be sure to configure the harakiri timeout and keep alive appropriately.
+
 ## Setup systemd
 
 We'll use `systemd` to control both uWSGI and Nautobot's background worker processes.
@@ -138,15 +147,15 @@ After=network-online.target
 Wants=network-online.target
 
 [Service]
-Type=forking
+Type=exec
 Environment="NAUTOBOT_ROOT=/opt/nautobot"
 
 User=nautobot
 Group=nautobot
-PIDFile=/var/tmp/nautobot-celery.pid
+PIDFile=/var/tmp/nautobot-worker.pid
 WorkingDirectory=/opt/nautobot
 
-ExecStart=/opt/nautobot/bin/nautobot-server celery worker --loglevel INFO --pidfile /var/tmp/nautobot-celery.pid
+ExecStart=/opt/nautobot/bin/nautobot-server celery worker --loglevel INFO --pidfile /var/tmp/nautobot-worker.pid
 
 Restart=always
 RestartSec=30
@@ -232,7 +241,39 @@ Once you've verified that the WSGI service and worker are up and running, move o
 
 ### SSL Error
 
-If you see the error `SSL error: decryption failed or bad record mac`, it is likely due to a mismatch in the uWSGI configuration and Nautobot's database settings.
-See [this conversation](https://github.com/nautobot/nautobot/issues/127) for more details.
+If you see the error `SSL error: decryption failed or bad record mac`, it is likely due to a mismatch in the uWSGI
+configuration and Nautobot's database settings.
 
 - Set `DATABASES` -> `default` -> `CONN_MAX_AGE=0` in `nautobot_config.py` and restart the Nautobot service.
+
+For example:
+
+```python
+DATABASES = {
+    "default": {
+        # Other settings...
+        "CONN_MAX_AGE": int(os.getenv("NAUTOBOT_DB_TIMEOUT", 0)),  # Change the value to 0
+    }
+}
+```
+
+Please see [SSL error: decryption failed or bad record mac & SSL SYSCALL error: EOF detected (#127)](https://github.com/nautobot/nautobot/issues/127) for more details.
+
+### Operational Error: Incorrect string value
+
+When using MySQL as a database backend, if you encounter a server error along the lines of `Incorrect string value: '\\xF0\\x9F\\x92\\x80' for column`, it is because you are running afoul of the legacy implementation of Unicode (aka `utf8`) encoding in MySQL. This often occurs when using modern Unicode glyphs like the famous poop emoji.
+
+- Create an entry in `DATABASES` -> `default` -> `OPTIONS` with the value `{"charset": "utf8mb4"}` in your `nautobot_config.py` and restart all Nautobot services. This will tell MySQL to always use `utf8mb4` character set for encoding.
+
+For example:
+
+```python
+DATABASES = {
+    "default": {
+        # Other setttings...
+        "OPTIONS": {"charset": "utf8mb4"},  # Add this line
+    }
+}
+```
+
+Please see [Computed fields with fallback value that is unicode results in OperationalError (#645)](https://github.com/nautobot/nautobot/issues/645) for more details.

nautobot/docs/plugins/development.md

@@ -434,8 +434,8 @@ graphql_types = [AnimalType]
 
 GraphQL utility functions:
 
-1) `execute_query()`: Runs string as a query against GraphQL.
-2) `execute_saved_query()`: Execute a saved query from Nautobot database.
+1. `execute_query()`: Runs string as a query against GraphQL.
+2. `execute_saved_query()`: Execute a saved query from Nautobot database.
 
 Both functions have the same arguments other than `execute_saved_query()` which requires a slug to identify the saved query rather than a string holding a query.
 
@@ -444,15 +444,15 @@ For authentication either a request object or user object needs to be passed in.
 Arguments:
 
 * `execute_query()`:
-  * query (str): String with GraphQL query.
-  * variables (dict, optional): If the query has variables they need to be passed in as a dictionary.
-  * request (django.test.client.RequestFactory, optional): Used to authenticate.
-  * user (django.contrib.auth.models.User, optional): Used to authenticate.
+    * query (str): String with GraphQL query.
+    * variables (dict, optional): If the query has variables they need to be passed in as a dictionary.
+    * request (django.test.client.RequestFactory, optional): Used to authenticate.
+    * user (django.contrib.auth.models.User, optional): Used to authenticate.
 * `execute_saved_query()`:
-  * saved_query_slug (str): Slug of a saved GraphQL query.
-  * variables (dict, optional): If the query has variables they need to be passed in as a dictionary.
-  * request (django.test.client.RequestFactory, optional): Used to authenticate.
-  * user (django.contrib.auth.models.User, optional): Used to authenticate.
+    * saved_query_slug (str): Slug of a saved GraphQL query.
+    * variables (dict, optional): If the query has variables they need to be passed in as a dictionary.
+    * request (django.test.client.RequestFactory, optional): Used to authenticate.
+    * user (django.contrib.auth.models.User, optional): Used to authenticate.
 
 Returned is a GraphQL object which holds the same data as returned from GraphiQL. Use `execute_query().to_dict()` to get the data back inside of a dictionary.
 

nautobot/docs/plugins/index.md

@@ -6,7 +6,7 @@ Plugins are packaged [Django](https://docs.djangoproject.com/) apps that can be
 
 The Nautobot plugin architecture allows for the following:
 
-* **Add new data models.** A plugin can introduce one or more models to hold data. (A model is essentially a table in the SQL database.)
+* **Add new data models.** A plugin can introduce one or more models to hold data. (A model is essentially a table in the SQL database.) These models can be integrated with core implmentations of GraphQL, webhooks, logging, custom relationships, custom fields, and tags.
 * **Add custom validation logic to existing data models.** A plugin can provide additional logic to customize the rules for validating created/updated data records.
 * **Add new URLs and views.** Plugins can register URLs under the `/plugins` root path to provide browsable views for users.
 * **Provide Jobs.** Plugins can serve as a convenient way to package and install [Jobs](../additional-features/jobs.md).
@@ -17,6 +17,7 @@ The Nautobot plugin architecture allows for the following:
 * **Add additional dependencies.** Custom Django application dependencies can be registered by each plugin.
 * **Declare configuration parameters.** Each plugin can define required, optional, and default configuration parameters within its unique namespace. Plug configuration parameter are defined by the user under `PLUGINS_CONFIG` in `nautobot_config.py`.
 * **Limit installation by Nautobot version.** A plugin can specify a minimum and/or maximum Nautobot version with which it is compatible.
+* **Add additional Git Providers.** Add additional Git Providers with a defined callback function to post process data received from the Git Repository.
 * **Register Jinja2 filters.** A plugin can define custom Jinja2 filters to be used in computed fields, webhooks, custom links, and export templates.
 
 ## Limitations

nautobot/docs/release-notes/version-1.1.md

@@ -22,7 +22,7 @@ While config contexts allow for arbitrary data structures to be stored within Na
 
 #### GraphQL ORM Functions
 
-Two new [GraphQL utility functions](../plugins/development.md) have been added to allow easy access to the GraphQL system from source code. Both can be accessed by using `from nautobot.core.graphql import execute_saved_query, execute_query`.
+Two new [GraphQL utility functions](../plugins/development.md#using-graphql-orm-utility) have been added to allow easy access to the GraphQL system from source code. Both can be accessed by using `from nautobot.core.graphql import execute_saved_query, execute_query`.
 
 1. `execute_query()`: Runs string as a query against GraphQL.
 2. `execute_saved_query()`: Execute a saved query from Nautobot database.
@@ -68,6 +68,31 @@ Please see the section on [migrating to Celery from RQ](../installation/services
 
 ### Removed
 
+## v1.1.0b2 (2021-07-09)
+
+### Added
+
+- [#599](https://github.com/nautobot/nautobot/issues/599) - Custom fields are now supported on `JobResult` objects
+- [#637](https://github.com/nautobot/nautobot/pull/637) - Implemented a `nautobot-server fix_custom_fields` command to manually purge stale custom field data
+
+### Changed
+
+- [#634](https://github.com/nautobot/nautobot/pull/634) - Documentation on plugin capabilities has been clarified.
+
+### Fixed
+
+- [#495](https://github.com/nautobot/nautobot/issues/495) - Fixed search for partial IPv4 prefixes/aggregates not finding all matching objects
+- [#533](https://github.com/nautobot/nautobot/issues/533) - Custom field tasks are now run atomically to avoid stale field data from being saved on objects.
+- [#554](https://github.com/nautobot/nautobot/issues/554) - Fixed search for partial IPv6 prefixes/aggregates not finding all matching objects
+- [#569](https://github.com/nautobot/nautobot/issues/569) - Change minimum/maximum allowed values for integer type in Custom Fields to 64-bit `BigIntegerField` types (64-bit)
+- [#600](https://github.com/nautobot/nautobot/issues/600) - The `invoke migrate` step is now included in the development getting started guide for Docker workflows
+- [#617](https://github.com/nautobot/nautobot/pull/617) - Added extra comments to `uwsgi.ini` config to help with load balancer deployments in Nautobot services documentation
+- [#626](https://github.com/nautobot/nautobot/pull/626) - Added prefix `NAUTOBOT_` in `override.env` example inside of `docker-entrypoint.sh`
+- [#645](https://github.com/nautobot/nautobot/issues/645) - Updated services troubleshooting docs to include "incorrect string value" fix when using Unicode emojis with MySQL as a database backend
+- [#653](https://github.com/nautobot/nautobot/issues/653) - Fixed systemd unit file for `nautobot-worker` to correctly start/stop/restart
+- [#661](https://github.com/nautobot/nautobot/issues/661) - Fixed `computed_fields` key not being included in API response for devices when using `opt_in_fields`
+- [#667](https://github.com/nautobot/nautobot/pull/667) - Fixed various outdated/incorrect places in the documentation for v1.1.0 release.
+
 ## v1.1.0b1 (2021-07-02)
 
 ### Added
@@ -78,9 +103,9 @@ Please see the section on [migrating to Celery from RQ](../installation/services
 - [#200](https://github.com/nautobot/nautobot/issues/200) - Jobs can be marked as read-only
 - [#274](https://github.com/nautobot/nautobot/issues/274) - Added config context schemas to optionally validate config and local context data against JSON Schemas
 - [#297](https://github.com/nautobot/nautobot/issues/297) -  Added an anonymous health-checking endpoint at `/health/`using, also introducing a `nautobot-server health_check` command.
-- [#485](https://github.com/nautobot/nautobot/pulls/485) - Applications can define navbar properties through `navigation.py`
+- [#485](https://github.com/nautobot/nautobot/pull/485) - Applications can define navbar properties through `navigation.py`
 - [#557](https://github.com/nautobot/nautobot/issues/557) - `Prefix` records can now be created using /32 (IPv4) and /128 (IPv6) networks. (Port of [NetBox #6545](https://github.com/netbox-community/netbox/pull/6545))
-- [#561](https://github.com/nautobot/nautobot/pulls/561) - Added autodetection of `mime_type` on `export_templates` provided by Git datasources
+- [#561](https://github.com/nautobot/nautobot/pull/561) - Added autodetection of `mime_type` on `export_templates` provided by Git datasources
 - [#636](https://github.com/nautobot/nautobot/pull/636) - Added custom fields to `JobResult` model, with minor changes to job result detail page
 
 ### Changed