VALIDATOR

Validator is an implementation of saltcheck written for pop and idem.

It is a test framework which enables running assertions on any idem execution modules. Like other pop projects, as additional capabilties are added or idem is extended to additional execution modules, so is validator. Like saltcheck, this capabiltiy can be used to validate idem states successfully ran, or to validate deployments, permissions, or anything else for which there is an execution module.

Usage Example

# validator examples.tst
test execution module ping:
    ----------
    status:
        Pass
    duration:
        0.0
test_args:
    ----------
    status:
        Pass
    duration:
        0.004
multiple assertion test:
    ----------
    assertion1:
        ----------
        status:
            Pass
    assertion2:
        ----------
        status:
            Pass
    assertion3:
        ----------
        status:
            Pass
    status:
        Pass
    duration:
        0.0025
test skip ping:
    ----------
    duration:
        0.0
    status:
        Skip
TEST RESULTS:
    ----------
    Passed:
        3
    Failed:
        0
    Skipped:
        1
    Execution Time:
        0.0105

tst Files

An idem test is written in a any rend supported syntax with a .tst extension. By default this is commonly used jinja and yaml.

Keywords

module_and_function:	(str) The idem execution module to be run.
args:	(list) Optional arguments passed to the execution module.
kwargs:	(dict) Optional keyword arguments to be passed to the execution module.
assertion:	(str) One of the supported assertions. Failed tests cause validator to exit with a non-zero exit code. Multiple assertions can be run against the output of a single module_and_function call. The assertion, expected_return, assertion_section, and assertion_section_delimiter keys can be placed in a list under an assertions key. See the multiple assertions example below.
expected_return:	(str) Required except by assertEmpty, assertNotEmpty, assertTrue, assertFalse. The return of module_and_function is compared to this value in the assertion.
assertion_section:	(str) Optional keyword used to parse the module_and_function return. If the execution module returns a list or dictionary as a result, the assertion_section value is used to lookup a specific value in that return for the assertion comparison.
assertion_section_delimiter:	(str) Optional delimiter to use when splitting a nested structure. Defaults to :
print_result:	(bool) Optional toggle to show values in test failure results. Defaults to True.
skip:	(bool) Optional toggle to skip running the individual test.

Supported Assertions

• assertEqual
• assertNotEqual
• assertTrue
• assertFalse
• assertIn
• assertNotIn
• assertGreater
• assertGreaterEqual
• assertLess
• assertLessEqual
• assertEmpty
• assertNotEmpty

Examples

Basic Test

test execution module ping:
  module_and_function: test.ping
  assertion: assertTrue

Jinja Example with assertion_section

{% set expected_value = 5 %}
test_jinja:
  module_and_function: cmd.run
  args:
    - echo
    - 6
  kwargs:
    shell: True
  assertion_section: stdout
  expected_return: {{ expected_value }}
  assertion: assertLessEqual

Multiple Assertions

multiple assertion test:
  module_and_function: cmd.run
  args:
    - echo
    - something
  assertions:
    - assertion: assertNotEmpty
    - assertion: assertEqual
      assertion_section: "stdout"
      expected_return: "something\n"
    - expected_return: s
      assertion: assertIn
      assertion_section: stdout

Skip

test skip ping:
  module_and_function: test.ping
  assertion: assertTrue
  skip: True