Add ability to store configuration settings in the database

[itdependsnetworks]

Environment

• Python version: 3.7
• Nautobot version: 1.0

Proposed Functionality

Provide some amount of settings via a singleton pattern.

Use Case

In developing the golden config plugin, I have settings that requires only a single instance in a database. Provide a pattern for singletons with configuration. This should include core, but a pattern for plugins too.

Database Changes

Not directly

External Dependencies

It could make sense to leverage a library, such as django-solo or django-constance.

[jathanism]

Hmm, I like this idea. It seems to me there are a lot of settings that could be stored in a database configuration that don't affect the operation of the service, that could be moved here. I'm thinking of settings like BANNER_TOP, BANNER_BOTTOM, MAINTENANCE_MODE, et al.

Could you please elaborate on your use-case so that we can consider how this might be implemented and how we can justify the means to the end?

[jvanderaa]

I'm thinking from the description that one of the models used in a plugin that I worked on would be able to benefit from it. Basically that there could only be one representation for each of the particular apps. I can explain what was done in some more detail as well

[glennmatthews]

Related to #541 in that we should provide a clear delineation between database storage of non-sensitive configuration data versus more secure ways of storing and managing sensitive data such as credentials and tokens.

[jedelman8]

Being how closely related to #580 this is, we should probably consolidate around this issue, but make it such that the plugin API can leverage patterns defined here (which is also stated in the description provided by @itdependsnetworks ).

@itdependsnetworks @jvanderaa we reviewed this last week and would like some feedback to drive the initial implementation.

Initially, it would not include storing any credentials in the "settings." That would stay as-is, e.g. in the config. This will change as we progress on #541

We also discussed limiting it to (a) basic key-value pairs more akin to environment variables (b) storing JSON vs. (c) modeling just what's possible in the custom field framework (so as that expands, this would as well).

Could you provide further thoughts and feedback based on your use cases?

[jvanderaa]

As I look at the future of plugins if every plugin has its settings in the UI there would be menu items for each of them. The thought is to have a consolidated settings page provided by Nautobot to house all of the settings for any plugin that would have settings. This way the UI is kept clean and there is a consistent method for providing the settings from plugin to plugin. Having 5 plugins, and 5 different settings pages will likely clutter the UI some.

[jedelman8]

Makes sense. We're really looking for the next level of feedback. It's what I was getting at here:

We also discussed limiting it to (a) basic key-value pairs more akin to environment variables (b) storing JSON vs. (c) modeling just what's possible in the custom field framework (so as that expands, this would as well).

Add a (d) future could even be a JSON Schema definition that generates JSON forms.

Can you comment on these options as viable options for plugins you've developed?

[jvanderaa]

I would be leaning towards (B) to have all of the settings available in the UI. The Onboarding Plugin stands out that would benefit being able to have a key-value mapping available for a sub setting, specifically in the platform map setting. (B) also covers (a) for the basic key value pairs.

[itdependsnetworks]

I had more advance use case then just a key/value store, such as linking to the a Git Repository. The following code will describe all of the actual use cases I have already had where I went with the singleton pattern:

https://github.com/nautobot/nautobot-plugin-golden-config/blob/d928ce9da926976f4ad96d99daabb0941ba66a20/nautobot_golden_config/models.py#L291-L367

Additionally, I could have pursued doing the following in the configuration settings as well: https://github.com/nautobot/nautobot-plugin-golden-config/blob/d928ce9da926976f4ad96d99daabb0941ba66a20/nautobot_golden_config/__init__.py#L21-L29

[glennmatthews]

It could make sense to leverage a library, such as django-solo or django-constance.

It looks like both of these libraries are oriented around managing settings via the Django admin views, for what it's worth.

[jedelman8]

Based on what we recently added within #768 with the plans to add JSON to CFs, modeling what we have for CFs for this seems to make sense. It give us text, integer, boolean, date, URL, selection, multiple selection with validation rules for settings.
In the short-term, that means we'd get all those options for settings (no JSON), but when JSON is added for CFs, it would be here as well. We'll need to get input from @nautobot/core too.

[itdependsnetworks]

Just in the interest of clarity, when I state Git Repository, I am referring to the Django ORM object type GitRepository, not a json serializable object.

[glennmatthews]

Brainstorming - global configuration (BANNER_TOP, BANNER_BOTTOM, etc.) probably would make sense to manage under django-admin (possibly with django-solo or django-constance), whereas plugin configs (#580) probably makes sense as a (single) menu item under the Plugins menu as we may want non-admin users to be able to manage plugin configuration.

[dgarros]

based on my experience, It's been useful to have BANNER_TOP, BANNER_BOTTOM defined in the configuration because we can assign it different value based on the environment Prod , Test, Dev etc ... and still easily import the database from prod to dev etc ..

[glennmatthews]

Fair point @dgarros! That's a general problem with this approach any storage of configuration in the DB - we'll be intermingling application configuration with application data, and there are any number of cases where that might not be desirable.

---

Thinking about this some more, a number of plugins (circuit-maintenance, chatops for examples) have configuration patterns that are better served by a dedicated database table, because they have a need for N configuration entries (access grants for chatops, notification sources for circuit-maintenance, etc) - these plugins would not benefit from a singleton design because it simply doesn't match how they need to be configured.

Similarly, settings like the onboarding plugin platform mappings, if desirable to store in the database, would IMHO be better implemented as a small dedicated table rather than as a JSON field in a singleton object.

For cases like the golden-config plugin, where there are a number of unique configuration keys, each with exactly one value, a singleton pattern for that specific plugin makes sense, but this would still typically be a different model/table for this plugin than for any other plugin.

In conclusion I find myself questioning what generic functionality in Nautobot core would make sense to add/include that wouldn't be better solved by a plugin-specific implementation that addresses the particular configuration needs of that plugin directly.

[jedelman8]

@dgarros do you have any opinion on what core parameters should be stored in the database?

[dgarros]

@jedelman8 it's a hard question, I can see only a few core parameters that could be useful to have in the database

RACK_ELEVATION_DEFAULT_UNIT_HEIGHT = 22
RACK_ELEVATION_DEFAULT_UNIT_WIDTH = 220

CHANGELOG_RETENTION = 90

HIDE_RESTRICTED_UI = False

MAINTENANCE_MODE = False

MAX_PAGE_SIZE = 1000
PAGINATE_COUNT = 50
PER_PAGE_DEFAULTS = [25, 50, 100, 250, 500, 1000]
PREFER_IPV4 = False

RELEASE_CHECK_URL = None
RELEASE_CHECK_TIMEOUT = 24 * 3600

[glennmatthews]

Thanks! Storing MAINTENANCE_MODE in the DB doesn't make sense I think (as once you set MAINTENANCE_MODE = True, the DB is read-only so you can never change it again... 😄 ) but those others definitely seem plausible to me.

[jathanism]

Two prominent solutions:

• django-solo - Simple and effective that solves the problem succinctly. You subclass SingletonModel with your own fields.
• django-constance - More comprehensive but more complicated. A project from everyone's favorite Jazz Band. Config. items can be provided inside of nautobot_config.py. Includes context processors and other extras.

I would say if we want a rigid data model for the configuration we go with django-solo, but if we want users to be able to add their own custom config items select django-constance.

[erjac77]

+1 for django-constance

We use it at work and we are happy with it.

[glennmatthews]

See #1079.