Operators for chained tests -- Fix #46

[eeishaan]

Fix #46

Glossary:
ETC: Elementary test condition

As proposed here, the grammar is as follows

tests:
          - exp

where
exp -> catA-op exp exp | catB-op ETC

catA-op -> AND | OR
catB-op -> NOT | epsilon

This grammar translates to the following example in yaml syntax:

tests:
- OR:
    - AND:
        - not-equal: last-state,Idle       # element in which test is performed
                err: "Test Failed!! last state is <{{post['last-state']}}>"
                info: "Test succeeded!! last state is not equal to idle, it is: <{{post['last-state']}}>"

        - NOT:
            - all-same: flap-count
                err: "Test Failed!!! flap count are not all same!!, it is <{{post['flap-count']}}> "
                info: "Test Succeeded!! flap count are all same, it is now <{{post['flap-count']}}>!!!"

   - is-equal: flap-count, 0
            err: "Test Failed!!! flap count is not equal to 0, it is: <{{post['flap-count']}}> "
            info: "Test Succeeded!! flap count is equal to <{{post['flap-count']}}> !!"

AND, OR are binary operators and NOT is a unary operator.
If any deviation is observed, for ex, passing more than one ETC to a unary operator or passing one ETC to a binary operator, the parser will simply ignore that instance and continue evaluating remaining sub-expressions.
If all sub-expressions are skipped then no ETCs are executed. For module version, this means that Operator.result will now be None instead of earlier True.

Changes to reporting:
Reporting works exactly as previously, nothing is removed only new things are added.
Now, each test's status is also shown in the final report.
For module version, one can access the new Operator.result_dict dictionary (key => test_name, value=> result[True/False/None]) to access the result of each test. User should ensure that test names in a test suite remain unique. If they repeat the result of older instance gets overwritten.
Also please note that one should not confuse the no. of test cases passed or fail reported at the end with the no. of sub-expressions failed or passed.

Example 1:

test.yml:

tests_include:
  - test_command_version

test_command_version:
  - command: show bgp neighbor
  - iterate:
      xpath: '/bgp-information/bgp-peer'
      id: peer-address
      tests:
          - OR:
            - NOT:
                - contains: //bgp-options, LogUpDown
            - all-same: flap-count
              err: "Test Failed!!! bgp-peer {{id_0}}!!! has mismatched flap count {{post['flap-count']}}"
              info: "Test Succeeded!! flap count is {{post['flap-count']}} for bgp-peer {{id_0}}!!!"
          - AND:
              - NOT:
                  - contains: //bgp-options, LogUpDown
                  - contains: //bgp-options, LogUpDown
              - is-equal: local-address, unspecified

Output:

jnpr.jsnapy.check: INFO: *************************** Device: 10.216.193.114 ***************************
jnpr.jsnapy.check: INFO: Tests Included: test_command_version 
jnpr.jsnapy.check: INFO: ************************* Command: show bgp neighbor *************************
jnpr.jsnapy.operator: INFO: PASS | All "//bgp-options" contains LogUpDown [ 9 matched ]
jnpr.jsnapy.operator: INFO: PASS | Value of all "flap-count" at xpath "/bgp-information/bgp-peer" is same [ 3 matched ]
jnpr.jsnapy.check: INFO: ERROR!!! Malformed sub-expression
jnpr.jsnapy.operator: INFO: PASS | All "local-address" is equal to "unspecified" [ 3 matched ]
jnpr.jsnapy.operator: INFO: ------------------------------- Final Result!! -------------------------------
jnpr.jsnapy.operator: INFO: test_command_version : Passed
jnpr.jsnapy.operator: INFO: Total No of tests passed: 3
jnpr.jsnapy.operator: INFO: Total No of tests failed: 0 
jnpr.jsnapy.operator: INFO: Overall Tests passed!!! 

Notice the error produced due to the malformed NOT sub-expression. The parser skipped the incorrect part and executed the rest ETCs and sub-expressions. As 3 of the ETCs were passed, the no of tests passed is reported to be so.

One thing to note: In cases where during the course of sub-expression evaluation, the number of valid sub-expressions drops below two for any of the binary operator, the operator will be rendered ineffective as

AND:
      - is-equal: local-address, unspecified

is meaningless. The ETC in question ( - is-equal: local-address, unspecified here ) will then become part of list of the parent operator.

Example 2:

test.yml

tests_include:
  - test_interfaces_terse

test_interfaces_terse:
  - command: show interfaces terse lo* 
  - item:
      id: ./name
      xpath: //physical-interface[normalize-space(name) = "lo0"]
      tests:
        - AND:  
            - is-equal: admin-status, down
              info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
              err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
            - is-equal: admin-status, up
              info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
              err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Output

Example 3:

test.yml

tests_include:
- test_interfaces_terse

test_interfaces_terse:
- command: show interfaces terse lo* 
- item:
   id: ./name
   xpath: //physical-interface[normalize-space(name) = "lo0"]
   tests:
     - OR:
         - AND:  
             - is-equal: admin-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: admin-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
         - OR:  
             - is-equal: oper-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: oper-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Output

Example 4:
test.yml

tests_include:
- test_interfaces_terse

test_interfaces_terse:
- command: show interfaces terse lo* 
- item:
   id: ./name
   xpath: //physical-interface[normalize-space(name) = "lo0"]
   tests:
     - OR:
         - AND:  
             - NOT:
               - is-equal: admin-status, up
                 info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
                 err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               - is-equal: admin-status, down
                 info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
                 err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
         - OR:  
             - is-equal: oper-status, down
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to down, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
             - is-equal: oper-status, up
               info: "Test Succeeded !! admin-status is equal to <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"
               err: "Test Failed !! admin-status is not equal to up, it is <{{post['admin-status']}}> with oper-status <{{pre['oper-status']}}>"

Output