Compliance examples & best usage

[PXC-BJ]

Hello,

I wanted to start by thanking the owner of this repo for the great plugin, I am looking at creating a number of compliance tests to see what capability this project can cover.

At present i am just doing something simple like, check interfaces ae0,ae1 etc for MTU > 9000

This works ok however i have a hard time debugging my tests while im creating them.

My question is twofold; is there are way to "print" or "echo" configuration back to the UI to see what configuration is present during test runtime? The reason for this is, if i write a test against 1000 hosts but 5 fail how do i know what devices failed the test and which configuration levels?

Second is there some more complex examples using dynamic data such as; as check for upstream connected neighbor (which from reading is taken from the netbox/DCIM data) to then double check we have the correct configuration for some other system, eg: NTP/Syslog etc.

I have also setup dynamic pairs for two hosts, we are trying to conduct a test to ensure configuration matches exactly between two given interfaces across both devices, the serialized configuration comes from TTP like this: (JSON)
{ "interfaces": { "ae10": [ { "mtu": 9212 }, { "physical_encapsulation": "flexible-ethernet-services", "units": { "101": [ { "logical_encapsulation": "vlan-bridge" }, { "vlan_id": 101 } ], "110": [ { "logical_encapsulation": "vlan-bridge" }, { "vlan_id": 110 } ], } } ], }

Selector is byName at the moment and we see both devices under the selector in the device tables.

We have written a test using the example guide here: https://validity.readthedocs.io/en/latest/features/dynamic_pairs/

However when we write a test to check there are no issues with the dynamic pairing like so:

device.dynamic_pair['interfaces']['ae10']

We see the below error:

TypeError: 'VDevice' object is not subscriptable

If there was a way to simple print what is being consulted at runtime it would be very helpful for constructing these tests.

Thank you

[amyasnikov]

Hi,

device.dynamic_pair['interfaces']['ae10']

dynamic_pair is a device object (VDevice to be precise), not dict. You should've written

device.dynamic_pair.config['interfaces']['ae10']

Second is there some more complex examples using dynamic data such as; as check for upstream connected neighbor

Yep, there is an example in the docs

https://validity.readthedocs.io/en/latest/features/howto_write_tests/#use-dcim-info-to-compare-connected-neighbors

Generally speaking, you can reach connected devices/interfaces using Django ORM queries like that one in the example, but usually it's not so easy due to the NetBox data model complexity.

is there are way to "print" or "echo" configuration back to the UI to see what configuration is present during test runtime?

Several approaches exist here. First of all, you can see the whole serialised config under the Serialized state device tab in the NetBox UI.
Secondly, each test result has an explanation table where you can see how the test expression was evaluated stage by stage. Thirdly, (it's a bit hacky, but anyway) if you have access to the shell where your NetBox is running, you can try using Django shell

cd /opt/netbox/netbox
./manage.py shell

from validity.models import VDevice

my_device = VDevice.objects.get(name="my_awesome_device")

print(my_device.config["interfaces"])

Btw I believe an AI agent like Codex or Claude Code can easily write you all the tests you want if it's provided with enough context (e.g. with articles from the docs and already working examples that you have).

[PXC-BJ]

Hi Amyasnikov,

Thank you for the swift reply, what i did try was this:

device.dynamic_pair.config['interfaces']['ae10']

However i received this error:

AttributeDoesNotExist: Attribute 'config' does not exist in expression 'device.dynamic_pair.config['interfaces']['ae10']'

I did initially ask Gemini to come up with some tests with varying successes, my MTU check for example initially went awry

I will absolutely check out the django shell you mention here and in the examples, I was straying away from this in attempt to keep my efforts to the GUI but i do have access to the host netbox runs on, when i did try entering this shell i hit this error:

`AttributeError: partially initialized module 'validity' has no attribute 'config' (most likely due to a circular import)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/opt/netbox/venv/lib/python3.12/site-packages/validity/init.py", line 5, in
from netbox.settings import VERSION
File "/opt/netbox/netbox/netbox/settings.py", line 886, in
raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: Plugin validity does not provide a 'config' variable. This should be defined in the plugin's init.py file and point to the PluginConfig subclass.
`

Which i was going to look at independently because it is as if the plugin/netbox (we use community ed) was not installed correctly

Appreciate your time in maintaining this project and answering my questions

[amyasnikov]

AttributeDoesNotExist: Attribute 'config' does not exist in expression 'device.dynamic_pair.config['interfaces']['ae10']'

This looks like a bug, please consider filing an issue with detailed steps of how to reproduce it

[amyasnikov]

@PXC-BJ You were right with device.dynamic_pair.config causing an exception. I easily reproduced it.
This bug is fixed in the new version.

[PXC-BJ]

Thank you @amyasnikov i will attempt to update our application to latest version and retest with dynamic pair options along with other points you mentioned

To do so can you confirm the version this is fixed in @amyasnikov ?

It appears we have 3.5.2 at the moment:

(venv) [netbox]$ pip list | grep valid
netbox-validity 3.5.2

[amyasnikov]

It's been fixed in 3.5.1

[PXC-BJ]

This is the error i hit in 3.5.2:

`Worker #0, Unhandled error occured: <class 'AttributeError'>: 'NoneType' object has no attribute 'slices'

File "/opt/netbox/venv/lib/python3.12/site-packages/validity/scripts/runtests/apply.py", line 151, in call
test_results = self.get_test_results(params, worker_id, executor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/validity/scripts/runtests/apply.py", line 163, in get_test_results
selector_devices = self.get_selector_devices(worker_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/netbox/venv/lib/python3.12/site-packages/validity/scripts/runtests/apply.py", line 172, in get_selector_devices
return job_extractor.parent.job.result.slices[worker_id]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

`

My test looks like this:

device.config['interfaces']['ae10'] == device.dynamic_pair.config['interfaces']['ae10']

My serialized state in devices for each device looks like this:

{
"interfaces": {
"ae10": [
{
"mtu": 9212
},
{
"physical_encapsulation": "flexible-ethernet-services",
"units": {

            etc

Let me know if i should raise a bug for this issue

Thanks again