Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ansible validate variable module #54089

Open
wants to merge 4 commits into
base: devel
from

Conversation

Projects
None yet
2 participants
@brutus333
Copy link

commented Mar 20, 2019

SUMMARY

This commit adds a new Ansible action plugin and corresponding module
that can be used to validate Ansible variables against JSON Schema
provided as an Ansible dictionary, Jinja2 expression or schema file.

ISSUE TYPE
  • New Module Pull Request
COMPONENT NAME

validate

ADDITIONAL INFORMATION

For complex data structures used in Ansible roles it is important to validate the input variables provided as inventory variables or extra vars since otherwise a large playbook may fail after running for a long time due to variable mismatch.
This allows a fast-fail behavior.

The validate module implements this validation using standard JSON schema implementations.

An added benefit of using JSON schema is the ability to document the data structure inside schema and use JSON to md or JSON to html converters to dynamically generate the documentation.


This commit adds a new Ansible action plugin and corresponding module
that can be used to validate Ansible variables against JSON Schema
provided as an Ansible dictionary, Jinja2 expression or schema file.
@ansibot

This comment has been minimized.

Copy link
Contributor

commented Mar 20, 2019

The test ansible-test sanity --test pylint [explain] failed with 9 errors:

lib/ansible/plugins/action/validate.py:89:43: bad-whitespace Exactly one space required after comma             from jsonschema import validate,draft7_format_checker,exceptions,RefResolver                                            ^
lib/ansible/plugins/action/validate.py:89:65: bad-whitespace Exactly one space required after comma             from jsonschema import validate,draft7_format_checker,exceptions,RefResolver                                                                  ^
lib/ansible/plugins/action/validate.py:89:76: bad-whitespace Exactly one space required after comma             from jsonschema import validate,draft7_format_checker,exceptions,RefResolver                                                                             ^
lib/ansible/plugins/action/validate.py:102:48: bad-whitespace Exactly one space required after comma                     self._load_files(self.schema,validate_extensions=True)                                                 ^
lib/ansible/plugins/action/validate.py:124:36: ansible-format-automatic-specification Format string contains automatic field numbering specification
lib/ansible/plugins/action/validate.py:129:44: bad-whitespace Exactly one space required after comma             validate(instance=variable_value,schema=schema_data,                                             ^
lib/ansible/plugins/action/validate.py:132:36: ansible-format-automatic-specification Format string contains automatic field numbering specification
lib/ansible/plugins/action/validate.py:132:87: bad-whitespace Exactly one space required after comma             raise AnsibleActionFail("Failed validating {} in {}: {}".format(e.validator,e.schema_path,e.message))                                                                                        ^
lib/ansible/plugins/action/validate.py:132:101: bad-whitespace Exactly one space required after comma             raise AnsibleActionFail("Failed validating {} in {}: {}".format(e.validator,e.schema_path,e.message))                                                                                                      ^

The test ansible-test sanity --test ansible-doc --python 2.6 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 2.7 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.5 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.6 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.7 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.8 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test docs-build [explain] failed with the error:

Command "/usr/bin/python test/sanity/code-smell/docs-build.py" returned exit status 1.
>>> Standard Error
Command 'make singlehtmldocs' failed with status code: 2
--> Standard Output
PYTHONPATH=../../lib ../bin/dump_config.py --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ -d ../../lib/ansible/config/base.yml
mkdir -p rst/cli
PYTHONPATH=../../lib ../bin/generate_man.py --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py
PYTHONPATH=../../lib ../bin/dump_keywords.py --template-dir=../templates --output-dir=rst/reference_appendices/ -d ./keyword_desc.yml
PYTHONPATH=../../lib ../bin/plugin_formatter.py -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/modules/ 
Evaluating module files...
Makefile:93: recipe for target 'modules' failed
--> Standard Error
Traceback (most recent call last):
  File "../bin/plugin_formatter.py", line 783, in <module>
    main()
  File "../bin/plugin_formatter.py", line 738, in main
    plugin_info, categories = get_plugin_info(options.module_dir, limit_to=options.limit_to, verbose=(options.verbosity > 0))
  File "../bin/plugin_formatter.py", line 294, in get_plugin_info
    doc, examples, returndocs, metadata = plugin_docs.get_docstring(module_path, fragment_loader, verbose=verbose)
  File "/root/ansible/lib/ansible/utils/plugin_docs.py", line 105, in get_docstring
    data = read_docstring(filename, verbose=verbose, ignore_errors=ignore_errors)
  File "/root/ansible/lib/ansible/parsing/plugin_docs.py", line 59, in read_docstring
    data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
  File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 35, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 111, in compose_sequence_node
    node.value.append(self.compose_node(node, index))
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 64, in compose_node
    if self.check_event(AliasEvent):
  File "/usr/local/lib/python3.6/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.6/dist-packages/yaml/parser.py", line 449, in parse_block_mapping_value
    if not self.check_token(KeyToken, ValueToken, BlockEndToken):
  File "/usr/local/lib/python3.6/dist-packages/yaml/scanner.py", line 115, in check_token
    while self.need_more_tokens():
  File "/usr/local/lib/python3.6/dist-packages/yaml/scanner.py", line 149, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/usr/local/lib/python3.6/dist-packages/yaml/scanner.py", line 289, in stale_possible_simple_keys
    "could not find expected ':'", self.get_mark())
yaml.scanner.ScannerError: while scanning a simple key
  in "<unicode string>", line 24, column 1:
    https://github.com/Julian/jsonsc ... 
    ^
could not find expected ':'
  in "<unicode string>", line 26, column 1:
    author:
    ^
make: *** [modules] Error 1

The test ansible-test sanity --test boilerplate [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: missing: from __future__ import (absolute_import, division, print_function)

The test ansible-test sanity --test pep8 [explain] failed with 10 errors:

lib/ansible/modules/utilities/logic/validate.py:34:161: E501 line too long (162 > 160 characters)
lib/ansible/modules/utilities/logic/validate.py:36:161: E501 line too long (210 > 160 characters)
lib/ansible/plugins/action/validate.py:89:44: E231 missing whitespace after ','
lib/ansible/plugins/action/validate.py:89:66: E231 missing whitespace after ','
lib/ansible/plugins/action/validate.py:89:77: E231 missing whitespace after ','
lib/ansible/plugins/action/validate.py:102:49: E231 missing whitespace after ','
lib/ansible/plugins/action/validate.py:127:9: E303 too many blank lines (2)
lib/ansible/plugins/action/validate.py:129:45: E231 missing whitespace after ','
lib/ansible/plugins/action/validate.py:132:88: E231 missing whitespace after ','
lib/ansible/plugins/action/validate.py:132:102: E231 missing whitespace after ','

The test ansible-test sanity --test validate-modules [explain] failed with 3 errors:

lib/ansible/modules/utilities/logic/validate.py:0:0: E319 RETURN.message.returned: required key not provided @ data['message']['returned']. Got None
lib/ansible/modules/utilities/logic/validate.py:38:1: E302 DOCUMENTATION is not valid YAML
lib/ansible/modules/utilities/logic/validate.py:47:20: E311 EXAMPLES is not valid YAML

The test ansible-test sanity --test yamllint [explain] failed with 2 errors:

lib/ansible/modules/utilities/logic/validate.py:38:1: error DOCUMENTATION: syntax error: could not find expected ':'
lib/ansible/modules/utilities/logic/validate.py:47:20: error EXAMPLES: syntax error: expected <block end>, but found '<scalar>'

click here for bot help

@ansibot

This comment has been minimized.

Copy link
Contributor

commented Mar 20, 2019

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/plugins/action/validate.py:124:36: ansible-format-automatic-specification Format string contains automatic field numbering specification
lib/ansible/plugins/action/validate.py:130:36: ansible-format-automatic-specification Format string contains automatic field numbering specification

The test ansible-test sanity --test ansible-doc --python 2.6 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 2.7 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.5 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.6 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.7 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test ansible-doc --python 3.8 [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:0:0: has a documentation error formatting or is missing documentation.

The test ansible-test sanity --test docs-build [explain] failed with the error:

Command "/usr/bin/python test/sanity/code-smell/docs-build.py" returned exit status 1.
>>> Standard Error
Command 'make singlehtmldocs' failed with status code: 2
--> Standard Output
PYTHONPATH=../../lib ../bin/dump_config.py --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ -d ../../lib/ansible/config/base.yml
mkdir -p rst/cli
PYTHONPATH=../../lib ../bin/generate_man.py --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py
PYTHONPATH=../../lib ../bin/dump_keywords.py --template-dir=../templates --output-dir=rst/reference_appendices/ -d ./keyword_desc.yml
PYTHONPATH=../../lib ../bin/plugin_formatter.py -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/modules/ 
Evaluating module files...
Makefile:93: recipe for target 'modules' failed
--> Standard Error
Traceback (most recent call last):
  File "../bin/plugin_formatter.py", line 783, in <module>
    main()
  File "../bin/plugin_formatter.py", line 738, in main
    plugin_info, categories = get_plugin_info(options.module_dir, limit_to=options.limit_to, verbose=(options.verbosity > 0))
  File "../bin/plugin_formatter.py", line 294, in get_plugin_info
    doc, examples, returndocs, metadata = plugin_docs.get_docstring(module_path, fragment_loader, verbose=verbose)
  File "/root/ansible/lib/ansible/utils/plugin_docs.py", line 105, in get_docstring
    data = read_docstring(filename, verbose=verbose, ignore_errors=ignore_errors)
  File "/root/ansible/lib/ansible/parsing/plugin_docs.py", line 59, in read_docstring
    data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
  File "/usr/local/lib/python3.6/dist-packages/yaml/constructor.py", line 35, in get_single_data
    node = self.get_single_node()
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 36, in get_single_node
    document = self.compose_document()
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 55, in compose_document
    node = self.compose_node(None, None)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 84, in compose_node
    node = self.compose_mapping_node(anchor)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 133, in compose_mapping_node
    item_value = self.compose_node(node, item_key)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 82, in compose_node
    node = self.compose_sequence_node(anchor)
  File "/usr/local/lib/python3.6/dist-packages/yaml/composer.py", line 110, in compose_sequence_node
    while not self.check_event(SequenceEndEvent):
  File "/usr/local/lib/python3.6/dist-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/usr/local/lib/python3.6/dist-packages/yaml/parser.py", line 393, in parse_block_sequence_entry
    "expected <block end>, but found %r" % token.id, token.start_mark)
yaml.parser.ParserError: while parsing a block collection
  in "<unicode string>", line 22, column 5:
        - 'This plugin uses JSONSchema p ... 
        ^
expected <block end>, but found '<scalar>'
  in "<unicode string>", line 27, column 33:
           the ansible plugin doesn't work with relative references.'
                                    ^
make: *** [modules] Error 1

The test ansible-test sanity --test validate-modules [explain] failed with 2 errors:

lib/ansible/modules/utilities/logic/validate.py:0:0: E319 RETURN.message.returned: required key not provided @ data['message']['returned']. Got None
lib/ansible/modules/utilities/logic/validate.py:40:33: E302 DOCUMENTATION is not valid YAML

The test ansible-test sanity --test yamllint [explain] failed with 1 error:

lib/ansible/modules/utilities/logic/validate.py:40:33: error DOCUMENTATION: syntax error: expected <block end>, but found '<scalar>'

click here for bot help

brutus333 added some commits Mar 20, 2019

@ansibot

This comment has been minimized.

Copy link
Contributor

commented Mar 20, 2019

@AnderEnder @gregswift @jarv @jhoekx @linuxdynasty @tbielawa

As a maintainer of a module in the same namespace this new module has been submitted to, your vote counts for shipits. Please review this module and add shipit if you would like to see it merged.

click here for bot help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.