Skip to content

Commit 797bfec

Browse files
lvrfrc87lvrfrc87
authored
Merge pull request #10 from networktocode-llc/lvrfrc87
parameter_match check type implementation
2 parents 283e999 + 3dfd83a commit 797bfec

File tree

5 files changed

+882
-10
lines changed

5 files changed

+882
-10
lines changed

‎README.md‎

Lines changed: 332 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,332 @@
1-
# netcompare# netcompre
1+
# netcompare
2+
3+
netcompare is a python library targeted at intelligently deep diffing structured data objects of different types. In addition, netcompare provides some basic tests of keys and values within the data structure. Ultimately, this library is meant to be a light-weight way to compare structured output from network device 'show' commands.
4+
5+
## Use Case
6+
7+
netcompare enables an easy and direct way to see the outcome of network change windows. The intended usage is to collect raw show command output before and after a change window. Prior to closing the change window, the results are compared to help determine if the change was successful and if the network is in an acceptable state. The output can be stored with the change's documentation for easy reference and proof of completion.
8+
9+
## Check Types
10+
11+
### exact_match
12+
13+
exact_match is concerned about the value of the elements within the data structure. The keys and values should match between the pre and post values.
14+
15+
```
16+
PASS
17+
--------------------
18+
pre: [{"A": 1}]
19+
post: [{"A": 1}]
20+
```
21+
22+
```
23+
FAIL
24+
--------------------
25+
pre: [{"A": 1}]
26+
post: [{"A": 2}]
27+
```
28+
29+
```
30+
FAIL
31+
--------------------
32+
pre: [{ "A": 1}]
33+
post: []
34+
```
35+
36+
### parameter_match
37+
38+
parameter_match provides a way to match keys and values in the output with known good values.
39+
40+
The test defines key/value pairs known to be the good value - type `dict()` - to match against the parsed output. The test FAILS if any status has changed based on what is defined in pre/post. If there are new values not contained in the input/test value, that will not count as a failure.
41+
42+
43+
Examples:
44+
45+
```
46+
{"A": 1, "B": 2}
47+
48+
PASS/PASS
49+
{"A": 1, "B": 2}
50+
{"A": 1, "B": 2}
51+
52+
PASS/PASS
53+
{"A": 1, "B": 2}
54+
{"A": 1, "B": 2, "C": 3}
55+
56+
PASS/FAIL
57+
{"A": 1, "B": 2}
58+
{"A": 1, "B": 666}
59+
60+
FAIL/PASS
61+
{"A": 1}
62+
{"A": 1, "B": 2}
63+
```
64+
65+
In network data, this could be a state of bgp neighbors being Established or the connectedness of certain interfaces being up.
66+
67+
### Tolerance
68+
69+
The `tolerance` test defines a percentage of differing `float()` between the pre and post checks. The threshold is defined as a percentage that can be different either from the value stated in pre and post fields.
70+
71+
The threshold must be `float > 0`, is percentge based, and will be counted as a range centered on the value in pre and post.
72+
73+
```
74+
Pre: 100
75+
Post: 110
76+
Threshold: 10
77+
-----------------
78+
PASS/PASS
79+
Pre: [100]
80+
Post: [110]
81+
82+
PASS/PASS
83+
Pre: [100]
84+
Post: [120]
85+
86+
PASS/PASS
87+
Pre: [100]
88+
Post: [100]
89+
90+
PASS/FAIL
91+
Pre: [100]
92+
Post: [90]
93+
94+
PASS/FAIL
95+
Pre: [90]
96+
Post: [20]
97+
98+
FAIL/FAIL
99+
Pre: [80]
100+
Post: [120]
101+
```
102+
103+
This test can test the tolerance for changing quantities of certain things such as routes, or L2 or L3 neighbors. It could also test actual outputted values such as transmitted light levels for optics.
104+
105+
## How To Define A Check
106+
107+
The check requires at least 2 arguments: `check_type` which can be `exact_match`, `tolerance`, `parameter_match` or `path`. The `path` argument is JMESPath based but uses `$` to anchor the reference key needed to generate the diff - more on this later.
108+
109+
Example #1:
110+
111+
Run an `exact_match` between 2 files where `peerAddress` is the reference key (note the anchors used - `$` ) for `statebgpPeerCaps`. In this example, key and value are at the same level.
112+
113+
Check Definition:
114+
```
115+
{
116+
"check_type": "exact_match",
117+
"path": "result[0].vrfs.default.peerList[*].[$peerAddress$,statebgpPeerCaps]",
118+
}
119+
```
120+
121+
Show Command Output - Pre:
122+
```
123+
{
124+
"jsonrpc": "2.0",
125+
"id": "EapiExplorer-1",
126+
"result": [
127+
{
128+
"vrfs": {
129+
"default": {
130+
"peerList": [
131+
{
132+
"linkType": "external",
133+
"localAsn": "65130.1100",
134+
"prefixesSent": 52,
135+
"receivedUpdates": 0,
136+
"peerAddress": "7.7.7.7",
137+
"v6PrefixesSent": 0,
138+
"establishedTransitions": 0,
139+
"bgpPeerCaps": 75759616,
140+
"negotiatedVersion": 0,
141+
"sentUpdates": 0,
142+
"v4SrTePrefixesSent": 0,
143+
"lastEvent": "NoEvent",
144+
"configuredKeepaliveTime": 5,
145+
"ttl": 2,
146+
"state": "Idle",
147+
...
148+
```
149+
Show Command Output - Post:
150+
```
151+
{
152+
"jsonrpc": "2.0",
153+
"id": "EapiExplorer-1",
154+
"result": [
155+
{
156+
"vrfs": {
157+
"default": {
158+
"peerList": [
159+
{
160+
"linkType": "external",
161+
"localAsn": "65130.1100",
162+
"prefixesSent": 50,
163+
"receivedUpdates": 0,
164+
"peerAddress": "7.7.7.7",
165+
"v6PrefixesSent": 0,
166+
"establishedTransitions": 0,
167+
"bgpPeerCaps": 75759616,
168+
"negotiatedVersion": 0,
169+
"sentUpdates": 0,
170+
"v4SrTePrefixesSent": 0,
171+
"lastEvent": "NoEvent",
172+
"configuredKeepaliveTime": 5,
173+
"ttl": 2,
174+
"state": "Connected",
175+
```
176+
177+
Result:
178+
```
179+
{
180+
"7.7.7.7": {
181+
"state": {
182+
"new_value": "Connected",
183+
"old_value": "Idle"
184+
}
185+
}
186+
}
187+
```
188+
`result[0].vrfs.default.peerList[*].$peerAddress$` is the reference key (`7.7.7.7`) that we want associated to our value used to generate diff (`state`)...otherwise, how can we understand which `statebgpPeerCaps` is associated to which `peerAddress` ?
189+
190+
Example #2:
191+
192+
Similar to Example 1 but with key and value on different level. In this example `peers` will be our reference key, `accepted_prefixes` and `received_prefixes` the values used to generate diff.
193+
194+
Check Definition:
195+
```
196+
{
197+
"check_type": "exact_match",
198+
"path": "global.$peers$.*.*.ipv4.[accepted_prefixes,received_prefixes]",
199+
}
200+
```
201+
202+
Show Command Output - Pre:
203+
```
204+
{
205+
"global": {
206+
"peers": {
207+
"10.1.0.0": {
208+
"address_family": {
209+
"ipv4": {
210+
"accepted_prefixes": -9,
211+
"received_prefixes": 0,
212+
"sent_prefixes": 0
213+
},
214+
....
215+
```
216+
217+
Show Command Output - Post:
218+
```
219+
{
220+
"global": {
221+
"peers": {
222+
"10.1.0.0": {
223+
"address_family": {
224+
"ipv4": {
225+
"accepted_prefixes": -1,
226+
"received_prefixes": 0,
227+
"sent_prefixes": 0
228+
...
229+
```
230+
231+
Result:
232+
```
233+
{
234+
"10.1.0.0": {
235+
"accepted_prefixes": {
236+
"new_value": -1,
237+
"old_value": -9
238+
}
239+
},
240+
...
241+
```
242+
243+
Example #3:
244+
245+
Similar to Example 1 and 2 but without a reference key defined in `path`, plus some excluded fields to remove verbosity from diff output
246+
247+
Check Definition:
248+
```
249+
{
250+
"check_type": "exact_match",
251+
"path": "result[*]",
252+
"exclude": ["interfaceStatistics", "interfaceCounters"],
253+
}
254+
```
255+
256+
Show Command Output - Pre:
257+
```
258+
{
259+
"jsonrpc": "2.0",
260+
"id": "EapiExplorer-1",
261+
"result": [
262+
{
263+
"interfaces": {
264+
"Management1": {
265+
"lastStatusChangeTimestamp": 1626247820.0720868,
266+
"lanes": 0,
267+
"name": "Management1",
268+
"interfaceStatus": "connected",
269+
"autoNegotiate": "success",
270+
"burnedInAddress": "08:00:27:e6:b2:f8",
271+
"loopbackMode": "loopbackNone",
272+
"interfaceStatistics": {
273+
"inBitsRate": 3582.5323982177174,
274+
"inPktsRate": 3.972702352461616,
275+
"outBitsRate": 17327.65267220522,
276+
"updateInterval": 300,
277+
"outPktsRate": 2.216220664406746
278+
},
279+
...
280+
```
281+
282+
Show Command Output - Post:
283+
```
284+
{
285+
"jsonrpc": "2.0",
286+
"id": "EapiExplorer-1",
287+
"result": [
288+
{
289+
"interfaces": {
290+
"Management1": {
291+
"lastStatusChangeTimestamp": 1626247821.123456,
292+
"lanes": 0,
293+
"name": "Management1",
294+
"interfaceStatus": "connected",
295+
"autoNegotiate": "success",
296+
"burnedInAddress": "08:00:27:e6:b2:f8",
297+
"loopbackMode": "loopbackNone",
298+
"interfaceStatistics": {
299+
"inBitsRate": 3403.4362520883615,
300+
"inPktsRate": 3.7424095978179257,
301+
"outBitsRate": 16249.69114419833,
302+
"updateInterval": 300,
303+
"outPktsRate": 2.1111866059750692
304+
},
305+
...
306+
```
307+
308+
Result:
309+
310+
```
311+
{
312+
"interfaces": {
313+
"Management1": {
314+
"lastStatusChangeTimestamp": {
315+
"new_value": 1626247821.123456,
316+
"old_value": 1626247820.0720868
317+
},
318+
"interfaceAddress": {
319+
"primaryIp": {
320+
"address": {
321+
"new_value": "10.2.2.15",
322+
"old_value": "10.0.2.15"
323+
}
324+
}
325+
}
326+
}
327+
}
328+
}
329+
```
330+
331+
See [test](./tests) folder for more examples.
332+

‎netcompare/check_type.py‎

Lines changed: 26 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
"""CheckType Implementation."""
22
from typing import Mapping, Tuple, Union, List
3-
from .evaluator import diff_generator
3+
from .evaluator import diff_generator, parameter_evaluator
44
from .runner import extract_values_from_output
55

66

@@ -18,10 +18,13 @@ def init(*args):
1818
return ExactMatchType(*args)
1919
if check_type == "tolerance":
2020
return ToleranceType(*args)
21+
if check_type == "parameter_match":
22+
return ParameterMatchType(*args)
23+
2124
raise NotImplementedError
2225

2326
@staticmethod
24-
def extract_value_from_json_path(
27+
def get_value(
2528
value: Mapping, path: Mapping, exclude: List = None
2629
) -> Union[Mapping, List, int, str, bool]:
2730
"""Return the value contained into a Mapping for a defined path."""
@@ -50,7 +53,12 @@ class ToleranceType(CheckType):
5053

5154
def __init__(self, *args):
5255
"""Tollerance init method."""
53-
self.tolerance_factor = float(args[1]) / 100
56+
try:
57+
tolerance = args[1]
58+
except IndexError as error:
59+
raise f"Tolerance parameter must be defined as float at index 1. You have: {args}" from error
60+
61+
self.tolerance_factor = float(tolerance) / 100
5462
super().__init__()
5563

5664
def evaluate(self, reference_value: Mapping, value_to_compare: Mapping) -> Tuple[Mapping, bool]:
@@ -73,10 +81,23 @@ def _within_tolerance(self, *, old_value: float, new_value: float) -> bool:
7381
return (old_value - max_diff) < new_value < (old_value + max_diff)
7482

7583

84+
class ParameterMatchType(CheckType):
85+
"""Parameter Match class implementation."""
86+
87+
def evaluate(self, reference_value: Mapping, value_to_compare: Mapping) -> Tuple[Mapping, bool]:
88+
"""Parameter Match evaluator implementation."""
89+
try:
90+
parameter = value_to_compare[1]
91+
except IndexError as error:
92+
raise f"Evaluating parameter must be defined as dict at index 1. You have: {value_to_compare}" from error
93+
diff = parameter_evaluator(reference_value, parameter)
94+
return diff, not diff
95+
96+
7697
# TODO: compare is no longer the entry point, we should use the libary as:
7798
# netcompare_check = CheckType.init(check_type_info, options)
78-
# pre_result = netcompare_check.extract_value_from_json_path(pre_obj, path)
79-
# post_result = netcompare_check.extract_value_from_json_path(post_obj, path)
99+
# pre_result = netcompare_check.get_value(pre_obj, path)
100+
# post_result = netcompare_check.get_value(post_obj, path)
80101
# netcompare_check.evaluate(pre_result, post_result)
81102
#
82103
# def compare(

0 commit comments

Comments
 (0)