Skip to content

yaml modules for working towards troubleshooting easement - #61183

Merged
Ch3LL merged 29 commits into
saltstack:masterfrom
whytewolf:yaml_helpers
Apr 6, 2022
Merged

yaml modules for working towards troubleshooting easement #61183
Ch3LL merged 29 commits into
saltstack:masterfrom
whytewolf:yaml_helpers

Conversation

@whytewolf

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds a yaml module and yamllint utils module to work towards adding tools to better help in troubleshooting salt.

What issues does this PR fix or reference?

#61182

Merge requirements satisfied?

[NOTICE] Bug fixes or features added to Salt require tests.

Commits signed with GPG?

No

Thomas Phipps added 4 commits November 2, 2021 21:23
eventually be yaml tools to help in troubleshooting yaml
functionality.
from __salt__.module.function to __salt__["module.function"] for the
test to work
@whytewolf
whytewolf requested a review from a team as a code owner November 3, 2021 21:23
@whytewolf
whytewolf requested review from joechainz and removed request for a team November 3, 2021 21:23
@github-actions

github-actions Bot commented Nov 5, 2021

Copy link
Copy Markdown
Contributor

Hi! I'm your friendly PR bot!

You might be wondering what I'm doing commenting here on your PR.

Yes, as a matter of fact, I am...

I'm just here to help us improve the documentation. I can't respond to
questions or anything, but what I can do, I do well!

Okay... so what do you do?

I detect modules that are missing docstrings or "CLI Example" on existing docstrings!
When I was created we had a lot of these. The documentation for these
modules need some love and attention to make Salt better for our users.

So what does that have to do with my PR?

I noticed that in this PR there are some files changed that have some of these
issues. So I'm leaving this comment to let you know your options.

Okay, what are they?

Well, my favorite, is that since you were making changes here I'm hoping that
you would be the most familiar with this module and be able to add some other
examples or fix any of the reported issues.

If I can, then what?

Well, you can either add them to this PR or add them to another PR. Either way is fine!

Well... what if I can't, or don't want to?

That's also fine! We appreciate all contributions to the Salt Project. If you
can't add those other examples, either because you're too busy, or unfamiliar,
or you just aren't interested, we still appreciate the contributions that
you've made already.

Whatever approach you decide to take, just drop a comment here letting us know!

Detected Issues (click me)
[INFO] Initializing environment for https://github.com/saltstack/pip-tools-compile-impersonate.
[INFO] Initializing environment for https://github.com/asottile/pyupgrade.
[INFO] Initializing environment for https://github.com/saltstack/pre-commit-remove-import-headers.
[INFO] Initializing environment for https://github.com/s0undt3ch/salt-rewrite.
[INFO] Initializing environment for https://github.com/timothycrosley/isort.
[INFO] Initializing environment for https://github.com/timothycrosley/isort:toml.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://github.com/asottile/blacken-docs.
[INFO] Initializing environment for https://github.com/asottile/blacken-docs:black==21.7b0.
[INFO] Initializing environment for https://github.com/PyCQA/bandit.
[INFO] Initializing environment for https://github.com/saltstack/invoke-pre-commit.
[INFO] Initializing environment for https://github.com/saltstack/invoke-pre-commit:blessings,distro,jinja2,msgpack,pyyaml.
[INFO] Initializing environment for https://github.com/saltstack/mirrors-nox.
[INFO] Initializing environment for https://github.com/saltstack/mirrors-nox:pip>=20.2.4,<21.2,setuptools<58.0.
[INFO] Installing environment for https://github.com/saltstack/invoke-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Check Known Missing Docstrings.......................(no files to check)Skipped
- hook id: invoke

Thanks again!

Comment thread salt/modules/yaml.py
Comment thread salt/modules/yaml.py
@whytewolf
whytewolf requested a review from Ch3LL February 24, 2022 23:47
@Ch3LL Ch3LL added the Phosphorus v3005.0 Release code name and version label Mar 3, 2022
@whytewolf

Copy link
Copy Markdown
Collaborator Author

rerun all

@Ch3LL

Ch3LL commented Apr 6, 2022

Copy link
Copy Markdown
Contributor

yay finally!!!

@Ch3LL
Ch3LL merged commit 062637a into saltstack:master Apr 6, 2022
@max-arnold

Copy link
Copy Markdown
Contributor

@whytewolf Hey! Trying the new yaml.lint function and it breaks in an obscure way. Ubuntu 20.04, apt install yamllint:

# phosphorus/foo.sls

foo:
  test.configurable_test_state:
    - changes: True
    - comment: A comment
salt-call yaml.lint salt://phosphorus/foo.sls

[ERROR   ] An un-handled exception was caught by Salt's global exception handler:
YamlLintConfigError: invalid config: unknown option "check-keys" for rule "truthy"
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3003rc1+2349.g87f6189e58', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/scripts.py", line 441, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/cli/call.py", line 50, in run
    caller.run()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/cli/caller.py", line 95, in run
    ret = self.call()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/cli/caller.py", line 202, in call
    ret["return"] = self.minion.executors[fname](
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/executors/direct_call.py", line 10, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/modules/yaml.py", line 56, in lint
    return __utils__["yamllint.lint"](yaml_out)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/utils/yamllint.py", line 49, in lint
    conf = YamlLintConfig(yamlconf)
  File "/usr/lib/python3/dist-packages/yamllint/config.py", line 43, in __init__
    self.validate()
  File "/usr/lib/python3/dist-packages/yamllint/config.py", line 121, in validate
    self.rules[id] = validate_rule_conf(rule, self.rules[id])
  File "/usr/lib/python3/dist-packages/yamllint/config.py", line 149, in validate_rule_conf
    raise YamlLintConfigError(
yamllint.config.YamlLintConfigError: invalid config: unknown option "check-keys" for rule "truthy"
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    load_entry_point('salt==3003rc1+2349.g87f6189e58', 'console_scripts', 'salt-call')()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/scripts.py", line 441, in salt_call
    client.run()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/cli/call.py", line 50, in run
    caller.run()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/cli/caller.py", line 95, in run
    ret = self.call()
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/cli/caller.py", line 202, in call
    ret["return"] = self.minion.executors[fname](
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/executors/direct_call.py", line 10, in execute
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/modules/yaml.py", line 56, in lint
    return __utils__["yamllint.lint"](yaml_out)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt-3003rc1+2349.g87f6189e58-py3.8.egg/salt/utils/yamllint.py", line 49, in lint
    conf = YamlLintConfig(yamlconf)
  File "/usr/lib/python3/dist-packages/yamllint/config.py", line 43, in __init__
    self.validate()
  File "/usr/lib/python3/dist-packages/yamllint/config.py", line 121, in validate
    self.rules[id] = validate_rule_conf(rule, self.rules[id])
  File "/usr/lib/python3/dist-packages/yamllint/config.py", line 149, in validate_rule_conf
    raise YamlLintConfigError(
yamllint.config.YamlLintConfigError: invalid config: unknown option "check-keys" for rule "truthy"

Am I doing something wrong?

@whytewolf

Copy link
Copy Markdown
Collaborator Author

@max-arnold must be a version issue. what version of yamllint did you end up with that doesn't have check-keys as one of the truthy rules?

for the record here is the config used to get as close to salt style yaml as possible

        extends: relaxed
        rules:
          line-length: { max: 256, level: warning }
          empty-lines: disable
          empty-values: {forbid-in-block-mappings: false, forbid-in-flow-mappings: true}
          trailing-spaces: disable
          key-ordering: disable
          truthy: {level: warning, check-keys: false }

@max-arnold

Copy link
Copy Markdown
Contributor

The yamllint version is the default one on Ubuntu 20.04: 1.20.0-1

@max-arnold

Copy link
Copy Markdown
Contributor

Maybe a version-dependent config should be implemented? Or if that is too much hassle, then a minimal version check at the module's __init__()?

@whytewolf

Copy link
Copy Markdown
Collaborator Author

@max-arnold Can you create a new issue for this? I"ll try and get around to it soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Phosphorus v3005.0 Release code name and version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants