feat: diode data plugin 1.0 rc #88

ltucker · 2025-04-23T20:07:40Z

This pull request introduces several updates to improve compatibility, enhance functionality, and clean up the codebase for the Diode NetBox plugin. Key changes include upgrading dependencies to support NetBox 4.2.3, simplifying plugin configuration by removing unused settings, and adding support for the netbox_branching plugin. The updates also include Docker configuration enhancements and workflow improvements.

Compatibility and Dependency Updates:

Updated the NetBox compatibility to require a minimum version of 4.2.3 and added support for version 1.0.0 of the plugin (netbox-plugin.yaml, netbox_diode_plugin/__init__.py). [1] [2]
Updated the Docker base image to netboxcommunity/netbox:v4.2.3-3.1.1 and added the netbox_branching plugin to dependencies (docker/Dockerfile-diode-netbox-plugin, docker/requirements-diode-netbox-plugin.txt). [1] [2]

Plugin Configuration Simplification:

Removed unused settings such as auto_provision_users, diode_to_netbox_username, and netbox_to_diode_username from the plugin configuration (README.md, netbox_diode_plugin/__init__.py). [1] [2]
Updated example configuration files to reflect the streamlined settings (docker/netbox/plugins_dev.py, docker/netbox/plugins_test.py). [1] [2]

Docker and Workflow Enhancements:

Added netbox-worker service to the Docker Compose configuration for background task processing and included health checks (docker/docker-compose.yaml).
Introduced a local_settings.py file to enable dynamic schema support with DynamicSchemaDict and a custom database router (docker/netbox/local_settings.py).
Updated GitHub Actions workflows to pin specific versions of actions for better reproducibility (.github/workflows/release.yaml, .github/workflows/lint-tests.yml). [1] [2]

Codebase Cleanup:

Refactored utility functions in configuration.py to use standard function definitions for better readability (docker/netbox/configuration/configuration.py).
Added a logging configuration to enable dynamic log levels based on the DEBUG environment variable (docker/netbox/configuration/logging.py).

Miscellaneous:

Updated copyright years to 2025 across multiple files (netbox_diode_plugin/__init__.py, netbox_diode_plugin/api/__init__.py). [1] [2]
Adjusted the .github/CODEOWNERS file to include @ltucker as a code owner.

* enable netbox_branching plugin by default in development/test docker image * include netbox-worker * add development setting defaults * don't pin branching plugin

* updates sdk to pick up ignored state * handle unknown states gracefully

* chore: security improvements (#53) * chore: add safe redirect to login Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * chore: gha - add missing permissions Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * chore: gha - pin actions to commit hashes Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * tidy up Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * update protobufs * update constants in tests --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> Co-authored-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

* chore: remove reconciler sdk Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix: docker setup with netbox 4.1.11 for netbox-branching plugin Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * feat: remove ingestion logs view Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * chore: update CODEOWNERS Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

* fix: retrieve object states using concrete models Search backend/CachedValue is global and doesn't seem to be reliable with branching Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * tidy up Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * chore: bump netbox min version and netbox-docker Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * chore: bump netbox min version Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix tests Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * use clean_fields() and tidy up Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix: add backwards compatible support for mac_address field (#62) * fix: add backwards compatible support for mac_address field * feat: extract and serialize site from scope Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * feat: apply change set - handle scope site Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> Co-authored-by: Luke Tucker <64618+ltucker@users.noreply.github.com>

* fix: retrieving object states of tags Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix: skip creating objects if already exist Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * chore: refactor apply change set view complexity Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * tidy up Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

* fix: scope support on apply change set (#64) Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * wip diff api * set default values and missing slugs Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * tidy up setting defaults Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * remove unused imports Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix constructor of object type Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * set slugs (if not present) after resolving existing instances Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * emit ref_id instead of variable object_id field for new objects * improve entity field mapping coverage * fill in primary value mapping, use primary value for slug * use canonical field ordering in change dicts * first pass at certain common circular refs * remove ref id to itself Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * tidy up Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * add applier Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix resolve ref before lookup, use field name directly, not field attr * don't query with unresolved references * fix _build_expressions_queryset Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * resolve lint issues Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * exclude fields with GenericRelation type Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix sorting dict we may get ints strings etc Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * rework applier logic Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * applier with content type fields Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix content type related existing value Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * exclude foreign key fields with many to one rel Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix: support for post create updates eg (primary mac address) (#68) * fix: support for post create updates eg (primary mac address) * use serializers, fix relevent tests * linting * filter fields in the prior state * add some basic smoke tests for diff / diff+apply * Change set validation (#69) * add change set validation Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * remove redundant serializers Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * remove redundant serializers tests Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * tidy up Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * adjust tests, lighter validation of refs on diff, handle generic --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> Co-authored-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> * fix: expand support for cycle breaking, add additional logical matchers (#70) * fix: fix error fingerprinting tags (#71) * fix: all noops -> no changes, show noops as only prior state (#72) --------- Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com> Co-authored-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

basic support for custom fields

cleanup error messages

special transformations to format inputs the way serializers expect adds handling for date only fields, decimal (vs float) and integer range

* fix: special handling for ip address network defaulting * fix: match ips ignoring mask value, use specific matchers

* fix: fix complex vm interface update case * lint

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

…x Plugin - Added DiodeOAuth2Authentication class for handling OAuth2 token validation. - Introduced IsDiodeOAuth2Authenticated permission class to check OAuth2 authentication. - Updated GenerateDiffView and ApplyChangeSetView to use the new authentication and permission classes. Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

…s files Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

* fix: handle callable defaults, defaults with non serializable values * fix: mark circuits.provider.accounts as circular * fix: error deduplicating device types * fix: report merge errors as 400 not 500 * fix: mark device oob_ip as circular * fix: adjust fingerprint matching for complex module bay case * fix: test virtual machine with primary_ip4 errors, logical cluster matching * fix: harmonize ZoneInfo values * fix: fix logical prefix in vrf lookup * fix: handle IntegrityError nicely * fix: support cached scope for matching * fix: add additional logical matchers * fix: don't alter arrays. output prior states for other ref types * fix: add table test for create/update of various types * fix: additional fixes from testing * fix: reduce logging, guard expensive trace logging with flag * fix: linting

- authentication with OAuth2 instead of API keys - squashed and removed old redundant migrations - removed redundant setup step Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

- remove unused imports Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

…lugin - Updated DiodeOAuth2Authentication to set user, token scopes, and token data on the request. - Introduced HasScope permission class to enforce required OAuth2 token scopes. - Refactored views to utilize IsAuthenticated and require_scopes for permission checks. - Adjusted unit tests to mock the updated authentication behavior. Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

feat: implement OAuth2 authentication and permissions

netbox_diode_plugin/api/plugin_utils.py

netbox_diode_plugin/api/transformer.py

netbox_diode_plugin/api/views.py

netbox_diode_plugin/api/plugin_utils.py

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

github-actions · 2025-04-23T23:51:21Z

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
2794	2600	93%	0%	🟢

New Files

File	Coverage	Status
netbox_diode_plugin/api/applier.py	92%	🟢
netbox_diode_plugin/api/authentication.py	96%	🟢
netbox_diode_plugin/api/common.py	88%	🟢
netbox_diode_plugin/api/differ.py	94%	🟢
netbox_diode_plugin/api/matcher.py	88%	🟢
netbox_diode_plugin/api/plugin_utils.py	82%	🟢
netbox_diode_plugin/api/supported_models.py	91%	🟢
netbox_diode_plugin/api/transformer.py	89%	🟢
netbox_diode_plugin/tests/test_api_diff_and_apply.py	100%	🟢
netbox_diode_plugin/tests/test_api_generate_diff.py	100%	🟢
netbox_diode_plugin/tests/test_authentication.py	99%	🟢
netbox_diode_plugin/tests/test_updates.py	93%	🟢
TOTAL	93%	🟢

Modified Files

File	Coverage	Status
netbox_diode_plugin/init.py	100%	🟢
netbox_diode_plugin/api/init.py	100%	🟢
netbox_diode_plugin/api/permissions.py	100%	🟢
netbox_diode_plugin/api/serializers.py	0%	🟢
netbox_diode_plugin/api/urls.py	100%	🟢
netbox_diode_plugin/api/views.py	79%	🟢
netbox_diode_plugin/forms.py	100%	🟢
netbox_diode_plugin/models.py	100%	🟢
netbox_diode_plugin/navigation.py	100%	🟢
netbox_diode_plugin/plugin_config.py	96%	🟢
netbox_diode_plugin/tests/init.py	100%	🟢
netbox_diode_plugin/tests/test_api_apply_change_set.py	99%	🟢
netbox_diode_plugin/tests/test_forms.py	100%	🟢
netbox_diode_plugin/tests/test_models.py	100%	🟢
netbox_diode_plugin/tests/test_plugin_config.py	100%	🟢
netbox_diode_plugin/tests/test_version.py	100%	🟢
netbox_diode_plugin/tests/test_views.py	100%	🟢
netbox_diode_plugin/urls.py	100%	🟢
netbox_diode_plugin/version.py	100%	🟢
netbox_diode_plugin/views.py	100%	🟢
TOTAL	94%	🟢

updated for commit: ece4fe8 by action🐍

ltucker and others added 30 commits December 12, 2024 12:14

feat: enable netbox_branching plugin by default (#51)

7838f9d

* enable netbox_branching plugin by default in development/test docker image * include netbox-worker * add development setting defaults * don't pin branching plugin

fix: do not treat _branch parameter as search criterion (#52)

3477b04

chore: update reconciler sdk to pick up ignored state (#54)

cd55ef8

* updates sdk to pick up ignored state * handle unknown states gracefully

chore: regenerate reconciler sdk (#56)

1fa65a4

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

chore: regenerate reconciler proto (#57)

89adbae

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

fix: scope support on apply change set (#64)

93e4c9b

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

fix: adjust result structure (#74)

fd79324

feat: basic support for custom fields (#75)

905e2c8

basic support for custom fields

fix: fix errors attempting value harmonization (#76)

a8019f9

cleanup error messages

fix: merge tags rather than overwrite (#77)

6275570

fix: accept both valid forms of protoJSON (snake and camel) (#78)

4ee1b5e

fix: add special format transformations (#80)

53158b2

special transformations to format inputs the way serializers expect adds handling for date only fields, decimal (vs float) and integer range

fix: special cases for ip network defaulting (#81)

9b01b3e

* fix: special handling for ip address network defaulting * fix: match ips ignoring mask value, use specific matchers

fix: fix complex vm interface update case (#82)

adf151f

* fix: fix complex vm interface update case * lint

fix: fix decimal / eui previous values (#83)

d847a00

fix: update dependency versions to allow for newer releases (#84)

bfe4788

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

chore: update copyright year in authentication, permissions, and view…

f0a8af4

…s files Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

feat: diode auth authentication

ce2bf8f

- authentication with OAuth2 instead of API keys - squashed and removed old redundant migrations - removed redundant setup step Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

chore: update compatibility version in netbox-plugin.yaml

305ddf3

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

feat: adjust unit test to diode auth with oauth2

8ea3b2b

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

chore: update copyright year in all plugin files to 2025

14149ac

- remove unused imports Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Merge branch 'feat-diode-data-plugin' into feat-authn-oauth2

99a1bb7

fix tests

35286bc

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

mfiedorowicz and others added 5 commits April 23, 2025 11:39

chore: remove unspecified diode_username_override

c2f3a85

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

refactor: rename permission scope constants for consistency

612c991

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

refactor: remove redundant HasScope permission class

6bd1215

Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>

Merge pull request #86 from netboxlabs/feat-authn-oauth2

9f3f219

feat: implement OAuth2 authentication and permissions

ltucker requested review from leoparente and mfiedorowicz as code owners April 23, 2025 20:07

github-actions bot added markdown dependencies python github-actions internal labels Apr 23, 2025

github-advanced-security bot found potential problems Apr 23, 2025

View reviewed changes

ltucker added 4 commits April 23, 2025 16:41

Merge branch 'develop' into feat-diode-data-plugin

421860f

fix: add rack role test

0bfd349

chore: remove trace logging

39a80a3

fix: sanitize branch name

ef1bee6

github-advanced-security bot found potential problems Apr 23, 2025

View reviewed changes

netbox_diode_plugin/api/plugin_utils.py Fixed Show fixed Hide fixed

ltucker and others added 3 commits April 23, 2025 17:03

Potential fix for code scanning alert no. 21: Log Injection

8c46fd2

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Potential fix for code scanning alert no. 16: Log Injection

3a70229

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

fix: fix flakey tag sorting, test cleanup

ece4fe8

mfiedorowicz approved these changes Apr 24, 2025

View reviewed changes

mfiedorowicz merged commit 28672d5 into develop Apr 24, 2025
6 checks passed

mfiedorowicz deleted the feat-diode-data-plugin branch April 24, 2025 07:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: diode data plugin 1.0 rc #88

feat: diode data plugin 1.0 rc #88

Uh oh!

ltucker commented Apr 23, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: diode data plugin 1.0 rc #88

feat: diode data plugin 1.0 rc #88

Uh oh!

Conversation

ltucker commented Apr 23, 2025

Compatibility and Dependency Updates:

Plugin Configuration Simplification:

Docker and Workflow Enhancements:

Codebase Cleanup:

Miscellaneous:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 23, 2025

☂️ Python Coverage

Overall Coverage

New Files

Modified Files

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants