feat(anta)!: Introduce AntaTest.Input and AntaTest.render()

[mtache]

Goal

1. Allow test developers to easily define test inputs
   a) Define test inputs in a single place (template parameters should not be exposed to test users)
   b) Provide better documentation for inputs
2. Provide input validation for tests using pydantic
3. Allow multiple templates per test
4. Error handling: the following situations must lead to a test result with status "error" (add documentation):
   • Input test validation (pydantic ValidationError for the AntaTest.Input model)
   • Exception in Anta.Test.render() user code
   • Error when running Anta.Test.collect()
   • Exception in AntaTest.test() user code
5. Reporting: add an option to hide tests in "error" status in reports
6. Tool: generate a test catalog example/documentation

Changes

• Introduce AntaTest.Input class.
  • This class is a pydantic BaseModel to define test inputs.
  • It is not mandatory for a test to define this class.
  • If defined, test inputs will be verified against the pydantic validation rules.
  • Inputs are accessible in all AntaTest methods using the instance attribute self.inputs.
• Introduce AntaTest.render(template: AntaTemplate) -> list[AntaCommand] method
  • Removed AntaTest.template class attribute.
  • For every AntaTemplate instance in AntaTest.commands class attribute, AntaTest.render() will be called.
  • In this method, test developers are expected to implement test inputs to template inputs mapping.
  • Test developers can choose to render multiple AntaCommand from a single AntaTemplate or not.

How to test

VerifyBGPIPv4UnicastCount and VerifyReachability has been implemented to support the new AntaTest definition.

Use the following test catalog:

---
anta.tests.routing:
  bgp:
    - VerifyBGPIPv4UnicastCount:
        vrfs:
            default: 3
            VRF10: 1
            VRF11: 1

anta.tests.connectivity:
    - VerifyReachability:
        hosts:
            - dst: 192.168.100.1
              src: 192.168.100.129
              vrf: MGMT
            - dst: 10.255.255.0
              src: 10.255.255.1

Design notes/ideas

• Today, AntaTest.instance_commands is a flatten list. This has the advantage to be a simple data structure but test developers need to deal with indexes instead of keys.
• As test inputs are now normalised using a pydantic model, unit tests could be simplified with one single function and parametrize it with ["test_class", "inputs", "eos_data", "expected_result", "expected_messages"]
• TestResult could carry an Exception object related to the test error. This would make easier to test ValidationError of AntaTest.Input models by using ValidationError.errors() instead of ValidationError.__str__(). Related question: is it necessary to test inputs validation as this is already implemented/tested in pydantic ?
• When there is an input validation error, we can skip the execution of render() in the constructor and more generally skip the instanciation of instance_commands attribute. However, the user can still call other methods of AntaTest like collect() or get some properties from the AntaTest object even if it does not make sense. We should implement an Exception for those cases.

Caveats

• # mypy: disable-error-code=attr-defined is required in the user test file definition because mypy does not manage to get the AntaTest.Input implementation in AntaTest subclasses: AntaTest.Input is an abstract class and can be implemented in AntaTest subclasses. self.inputs is an instance attribute allocated in the AntaTest constructor. When using self.inputs in AntaTest subclasses, mypy thinks AntaTest.Input typing is defined in AntaTest and not its subclasses.
• When using TypedDict as a pydantic type, there are caveats with Python 3.8: https://docs.pydantic.dev/2.0/usage/types/dicts_mapping/#typeddict

TODO

• Generate test documentation from input models with mkdocs -> Will be done in another PR
• Refactor all tests in anta.tests
• Write unit tests for new AntaTest behaviour
• Rewrite the documentation on custom tests

[gmuloc]

on the principle it looks very good to me :) - would be nice to see the rendered documentaiton and if we can do something for the test inputs in mkdocs to make them nicely presented!

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[titom73]

Seems a is missing in the PR tests.lib.test_case

Error reproduced in both:

• CI
• Local execution

Error message

________________________________________________________________________________________ ERROR collecting tests/units/anta_tests/test_aaa.py _________________________________________________________________________________________
ImportError while importing test module '/home/tom/Projects/arista/anta/tests/units/anta_tests/test_aaa.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../.pyenv/versions/3.9.9/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/units/anta_tests/test_aaa.py:17: in <module>
    from tests.lib.test_case import test  # noqa: F401
E   ModuleNotFoundError: No module named 'tests.lib.test_case'

Repository structure

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[gmuloc]

let's ship it !