Introduce "module.xrun"

[isbm]

What does this PR do?

Adds a feature: module.xrun for states.

Motivation

Current way of calling Salt modules with module.run has pretty cumbersome interface along with the fact it has also reserved words. This leads to a strange states, something like:

show_friends:
  module.run:
    - name: facebook.list_friends
    - m_name: thatch

Or even worse:

list_states:
  module.run:
    - name: united.states
    - m_states: Florida, Arizona, California
    - m_names: Tallahassee, Phoenix, Sacramento
    - m_fun: Bowling
    - kwargs: {
      'drink_beer': True,
      'visit': [
        'Florida State Capitol',
        'Arizona Science Center',
        'California Railroad State Museum'
      ],
      'call': thatch
    }

This led to the reasoning make it easier and simpler to use, throw away m_ prefix (this just sucks, sorry). Also write better unit tests. 😄 So the above can be done something like this:

show_friends:
  module.xrun:
    facebook.list_friends:
      - name: thatch

Look, Ma! No m_ prefix! How about another example:

list_states:
  module.xrun:
    united.states:
      - states: Florida, Arizona, California
      - names: Tallahassee, Phoenix, Sacramento
      - fun: Bowling
      - drink_beer: True
      - visit:
        - Florida State Capitol
        - Arizona Science Center
        - California Railroad State Museum
      - call: thatch

One more thing you can do (optionally, of course, you don't have to):

this_calls_it_at_once:
  module.xrun:
    some.function:
      - foo
      - bar
    another.function:
      - wow
      - that's handy

😎

Tests written?

Yes

[thatch45]

This looks great, I never liked the module.run interface. I think this is significantly cleaner

[isbm]

@thatch45 thanks! :) Before @cachedout pointed me to the Lint, I am already fixing it!!

[gtmanfred]

👍

[isbm]

@cachedout I do not understand Lint error of W1699 against zip(... and on top of the source. It works just fine in Python3 and Python2. What exactly is wrong and how to fix it?

[rallytime]

Very nice! This will be great to have.

I only have one small request: Can you add this new option to the Nitrogen Release Notes? I think people will find this useful and will want to know about it!

[whiteinge]

Might be worth adding a deprecation notice to module.run for this release or the next.

Given how heavily used it is we might not want to actually put it on a deprecation path but rather just add a notice to that docstring -- the 'ol file.sed treatment. 😉

[isbm]

@rallytime I added release notes and hopefully it is correct.

@whiteinge I added decorator-based deprecation (and also fixed it is now working for state modules as well). Now it will simply be using new module.run, but you can always restore an old behaviour with the use_deprecated mechanism (and have warning log messages about it every time you calling that). I mentioned all that in examples within the release notes. Complete implementation removal of old module.run is targeting Oxygen. We still got lots of time for it.

Hint for packagers: create a module_run_deprecation.conf file with the configuration mentioned in the release notes and deploy it in your minion package (RPM or deb etc) as a post-install script (so the file doesn't really belong to the package). Then later, when your infrastructure has been migrated, simply delete that config everywhere and restart minions.

[isbm]

@cachedout by now all the issues were addressed. If you think those two lints are just misbehaviour of custom scripts (at my side it doesn't hurt for Python 3 actually), then we are clean to merge.

[whiteinge]

@isbm sorry my comment above wasn't very clear.

I strongly think that we should not deprecate the old module.run. It's been around for far too long and is used in the wild far too much. The "file.sed treatment" is adding a note to the docstring of the old function that says, "Hey, you should use other function instead," but keeping the old one in the codebase more or less indefinitely.

Thoughts from the Core team? Agree/disagree?

[isbm]

@whiteinge But this is what exactly is happening. Add to your minion config:

use_deprecated:
  - module.run

And you will have the old one. All is needed to be done is to package it properly. E.g. we will ship something like /etc/salt/minion.d/module_run_deprecated.conf with the content above. Then you will see those messages in log and you can just "fix" the syntax eventually. Because having both that does the same is kind of pollution (in my eyes). Having that renamed is to lose "run" name, which nothing can be better.

So let's just slowly migrate the world. Two years should be enough to write - foo: bar instead of {foo: 'bar'}. 😄

[thatch45]

Both good points, I think that slowly moving the world would be a good idea, we could easily make sure that the comment returned from module.run specifies that it is going to be deprecated...

[whiteinge]

I can't speak directly to pros/cons of relying on package upgrades to put that config file in place. Personally, I'd prefer an opt-in rather than an opt-out for such a core module.

That said, I'll defer to the Core team since they will be on the front lines of any user upgrade problems.

[thatch45]

I would also prefer for opt-in, we do not have the luxury that SUSE has of dictating how the platform is going to be used unfortunately :(

[isbm]

@whiteinge @thatch45 no problems for opt-in. I will change that in the entire mechanism, so all the deprecation will work always on opt-in with "nag messages" to the log. Good point! But now is 1:00AM in Germany, so let me fix that tomorrow CEST. :-)

[isbm]

@thatch45 @whiteinge specially for your preference! Have a nice weekend, guys.

P.S. Hmm, nice deprecation mechanism we've just got, BTW...

[ScoreUnder]

Could potentially overwrite the ret['comment'] & ret['result'] from a previous loop iteration where it fails, depending on the content of the SLS file and the order of the dict.

This would mask a failure under some circumstances when test=True.

[isbm]

Oh, knapp, you're right. And I forgot to update docstring. Give me a sec...

[isbm]

@ScoreUnder done!

[terminalmage]

I think the config options can be better retrieved using config.get. There are also a number of things I noticed in the RST that we can clean up.

[terminalmage]

Should be "has dropped".

[terminalmage]

This can be expressed better in RST using:

.. note::
    It is necessary to explicitly turn on the new behavior (see below)

[terminalmage]

@thatch45 Did we ever make a firm decision on whether to use US or international spelling for words ending in -or/-our?

[isbm]

Not american here, sry.

[terminalmage]

I honestly don't care which we use, but I think that we should choose one for consistency across the docs.

[cachedout]

They are mixed in the docs. I agree we should pick a style but since we haven't, I see no reason to block on this.

[terminalmage]

In that case, yes I definitely agree.

[terminalmage]

This should be

Please keep in mind that the old syntax will no longer be supported in the ``Oxygen`` release of Salt.

[terminalmage]

The note here could also be put in a .. note:: RST block.

I don't understand this note though, to be honest. Are you suggesting that all minions should have their config files edited? Have you considered using config.get to get these configuration options? It checks grains and pillar as well as the minion config file. This would allow for this option to be turned on simply by editing the pillar data (or using grains.setval to set the value in the minion's /etc/salt/grains). Both of these options seem less intrusive IMO than editing all minions' config files.

[isbm]

@terminalmage good point on grains. I will extend that, but with a different PR, since I would like to have a different tests for it.

[terminalmage]

This would read better as:

To enable the new behavior, add the following to the minion config file:

[cachedout]

Final call for comments on this one. Otherwise, everything is looking good for a merge.

[rallytime]

@cachedout What do you want to do about the PY3 lint errors for the changes in module.py?

https://jenkins.saltstack.com/job/PR/job/salt-pr-lint-n/9449/violations/file/salt/states/module.py/

[isbm]

@rallytime I already asked this 5 days ago: please advice what it does mean. Because the code is not incompatible with Python 3 as lint claims. I would think that the script/lint check is flawed, as it does not provide even the reason and proposed fix. Just complaining and complaining wrong (unless I am missing something).

Suggestions? I can fix it in 5 minutes, but if I know what...

[rallytime]

@isbm Right, all good points.

For the record, and so everyone doesn't have to click around and find it, the console output for the PY3 violation displays the following:

16:14:34 salt/states/module.py:183: [W1699(incompatible-py3-code), ] Incompatible Python 3 code found. Proposed fix:
16:14:34  from salt.ext.six.moves import range
16:14:34  from salt.exceptions import SaltInvocationError
16:14:34  from salt.utils.decorators import with_deprecated
16:14:34 +from salt.ext.six.moves import zip
16:14:34  
16:14:34  
16:14:34  def wait(name, **kwargs):
16:14:34 salt/states/module.py:311: [W1699(incompatible-py3-code), ] Incompatible Python 3 code found. Proposed fix:
16:14:34      :return:
16:14:34      '''
16:14:34      argspec = salt.utils.args.get_function_argspec(__salt__[name])
16:14:34 -    func_kw = dict(zip(argspec.args[-len(argspec.defaults or []):], argspec.defaults or []))
16:14:34 +    func_kw = dict(list(zip(argspec.args[-len(argspec.defaults or []):], argspec.defaults or [])))
16:14:34      func_args = []
16:14:34      for funcset in kwargs.get('func_args') or {}:
16:14:34          if isinstance(funcset, dict):

But, as you stated, this should be working in both versions. If this is something wrong with our linter, then we need to address that.

I am thinking @s0undt3ch or @cachedout should provide some direction on this.

[isbm]

@rallytime Hmm. I don't think this is needed fix: the only difference that zip in Python3 returns iterator instead of list. This fix is trying to replicate the behaviour in a sense that it first turns iterator to the list. I personally wouldn't do that, because iterators are better. And I still think that the message is still misleading, if by "incompatible" we normally understand "unable to compile".

[isbm]

@cachedout OK, today I re-based it one more time against origin/develop for you (this morning and just now). Lint errors at least gone and unit tests works. What else I should do?

[cachedout]

@isbm That's all I think we need. If the tests pass, I'll merge it.

[isbm]

@cachedout it is all @rallytime 's fault. 😆 (no, not really: thanks for adding unit tests for with_deprecated which supposed to be mine technical debt!). I fixed them + added few more.