Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Got Ansible own modules? Run them from within the Salt just like that! #43504

Merged
merged 47 commits into from Nov 13, 2017

Conversation

Projects
None yet
4 participants
@isbm
Copy link
Contributor

commented Sep 14, 2017

What does this PR do?

In case you want to migrate away from Ansible, but have already modules written for it — no worries. Just toss them in and run them from Salt and Salt states.

Limitations

Currently, only Python-written modules are supported. Support for other languages are still in progress.

Requirements

The entire Ansible should be installed on the Minion. No remote execution at the moment. Continuation of this work can pull required files from the Ansible through the Salt transport. These files should be installed on Master only (in a future) and then thus get executed on the Minions.

Description and Usage

Normally Salt modules are usually module.function. That is, a typical call of a function from the module will be: salt hostname module.function some=parameters. But since we are connecting the entire Ansible world to the Salt universe, we would like to keep the namespace as this is inside the Ansible. That said, any Ansible module is placed in the sub-directory that potentially can clash with other same-named module. Regardless the future plans of Ansible community, we already have a namespace introduced. Therefore any Ansible module is called with a full "path", prefixed with ansible:

salt \* ansible.system.ping data="Hello from Ansible via Salt"

If you use typical development environment, this will result to the following:

saltdevel:
    ----------
    ping:
        Hello from Ansible via Salt

Listing available Ansible modules

In order to list all available Ansible modules, call ansible.list:

salt yourminion ansible.list

To filter them out, just type a string that matches inside them, e.g. "linux":

$ salt saltdevel ansible.list info
saltdevel:
    - net_tools.ipinfoio_facts
    - network.cloudengine.ce_info_center_debug
    - network.cloudengine.ce_info_center_global
    - network.cloudengine.ce_info_center_log
    - network.cloudengine.ce_info_center_trap

Getting more information about the module

You can read all the details about any Ansible modules by issuing ansible.help. For example:

$ salt saltdevel ansible.help net_tools.ipinfoio_facts
saltdevel:
    ----------
    Available sections on module "net_tools.ipinfoio_facts":
        - version_added
        - author
        - short_description
        - notes
        - options
        - module
    Description:
        - Gather IP geolocation facts of a host's IP address using ipinfo.io API

At this point you may ask for short_description, notes etc. For example, to know how to use it, call options:

$ salt saltdevel ansible.help net_tools.ipinfoio_facts options
saltdevel:
    ----------
    options:
        ----------
        http_agent:
            ----------
            default:
                ansible-ipinfoio-module/0.0.1
            description:
                - Set http user agent
            required:
                False
        timeout:
            ----------
            default:
                10
            description:
                - HTTP connection timeout in seconds
            required:
                False

Calling Ansible modules

Calling Ansible modules are just like any other Salt modules, straight-forward, except they will be needing to stay within their own namespace, which is different than we are using typically in Salt:

$ salt saltdevel ansible.net_tools.ipinfoio_facts                                    
saltdevel:
    ----------
    ansible_facts:
        ----------
        city:
            Nuremberg
        country:
            DE
        hostname:
            charybdis-ext.suse.de
        ip:
            195.135.221.2
        loc:
            49.4478,11.0683
        org:
            AS29298 SUSE LINUX GmbH
        postal:
            90409
        region:
            Bavaria
    changed:
        False

Calling from the SLS files

That is also supported. Naturally, you can use typical module.run if you like, but you've also got a dedicated Ansible state module. The syntax/example is the same as for new syntax for module.run. It also supports a "batch mode", or "Ansible playbook mode" where tasks can be executed from within one SLS file:

task_id:        # ID of your task
  ansible.call:  # Mandatory. This calls the modules
    - namespace.module:
      - params
    - namespace.another.module:
      - params
      - some: keywords

For example, this state will fail, because foo.bar module does not exist, as well as network.junos.junos_package requires some parameters. However it will still call everything one after another:

run_test_echo:
  ansible.call:
    - system.ping:
        - data: "Hello from Salt"
    - network.junos.junos_package:
    - foo.bar:

This results to the following output:

$ salt saltdevel state.apply ans
saltdevel:
----------
          ID: run_test_echo
    Function: ansible.call
      Result: False
     Comment: 
     Started: 15:33:04.367029
    Duration: 1409.689 ms
     Changes:   
              ----------
              foo.bar:
                  Module "foo.bar" failed. Error message: (KeyError) 'ansible.foo.bar'
              net_tools.ipinfoio_facts:
                  ----------
              network.junos.junos_package:
                  ----------
                  failed:
                      True
                  msg:
                      missing required arguments: src
              system.ping:
                  ----------
                  ping:
                      Hello from Salt

Summary for saltdevel
------------
Succeeded: 0 (changed=1)
Failed:    1
------------
Total states run:     1
Total run time:   1.410 s

Tests written?

Yes

Please review Salt's Contributing Guide for best practices.

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Sep 14, 2017

@meaksh @dincamihai @cachedout — guys, Merry Christmas. 😉 Keep an eye on it (ignore lints and little bits, let it boil, I will look after a while).
@thatch45 Just like promised. 😎

@isbm isbm force-pushed the isbm:isbm-ansible-module-dev branch 2 times, most recently from 4239191 to 43b0c48 Sep 15, 2017

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Sep 15, 2017

re-run lint

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Sep 15, 2017

This is awesome!

My main concern here is the departure from the salt_module.salt_function namespace. I think we could still maintain our current standard and just call Ansible modules with something like ansible.call ansible_module.ansible_function.

How do you feel about that, @isbm ?

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Sep 16, 2017

@cachedout I feel not very well about. 😉 We actually thought about that and expected this concern, still chose to depart from that for these reasons:
— Ansible is placing everything under different directories. While it currently maintains distinct names, yet their entire structure more like objects or at least namespaced structure. That said, nobody stops any user have two modules with the same name in different places. Since Ansible module does only one thing at a time, this problem might arise pretty much soon. Moreover, these modules are from kind of "another world", where keeping them isolated is good for two another reasons: a) Users mainly will reuse their Ansible modules from Salt in order not to rewrite them. They can happily place their same-name modules in the different directories for more convenient handling etc. b) We don't need to worry for name clashes. That said, at least keeping the namespace seems to be something that is rather needed.
— The implementation option that does something like ansible.call 'some.ansible.module' looks to me pretty messy. It is better and cleaner to do Python way, e.g. in import foo.bar.blah rather then go Perl/Ruby/PHP way: require "foo.bar.blah" where that "require" would resembled "ansible.call". Another problem with this, that it gives you a feeling that the whole thing is kind of away from the Salt, while it isn't. In fact, the whole exercise is to bring the Ansible modules into the Salt, not call Ansible modules outside the Salt! And this is very important part here.
— The code is much smaller then. 😉 Basically, it introspects and installs callers into the module directly "as is", instead of keeping them away, find out, call etc.
— It is still better to have in a future:

   salt \* docker.v1.create ....
   salt \* docker.v2.create ....
   salt \* docker.v3.create ....

...rather then

   salt \* docker.create ...
   salt \* docker_ng.make ...
   salt \* docker_reallygoodthistime.construct ...
   salt \* docker_even_better.build ...

😆

On the other hand, why not depart anyway? Modules in Salt are module.function, OK. But what possibly can go wrong with module.class.function or module.submodule.function anyway as long as we have a good introspection where anyone can call "help" over "class/submodule" and see what functions are there? I find this an advantage that should not be overused, sure, but having it when it is needed isn't really a problem. Especially that Ansible caller already lists you all modules with the namespaces. You can just copypaste that and it is there.

Alas, when the entire community will still start cry "Nah, no, we do not want stick to the Salt spirit and do command target module.function options (so to say: salt \* ansible.foo.bar.spam some=options) but we love to type really long commands such as command target module.callerfunction ofwhichmodule options (so to say salt \* ansible.callmethat "foo.bar.spam" some=options) then we can always go there as well (with onion behind the cheek, crying). 😉

@isbm isbm force-pushed the isbm:isbm-ansible-module-dev branch from 43b0c48 to d6c4944 Sep 18, 2017

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Sep 18, 2017

Personally I do not like the departing from the module.function model. We already have other similar things that work with the whatever.call model.

Between the salt module

https://docs.saltstack.com/en/latest/ref/runners/all/salt.runners.salt.html#salt.runners.salt.cmd

And the way that most proxy minions are written.

https://github.com/saltstack/salt/blob/2017.7/salt/modules/esxi.py#L52

We already have this kind of behavior in place.

My 2¢, Thanks.

Daniel

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Sep 19, 2017

@gtmanfred It is not a runner module, and the paradigm is different. When an Ansible module is called, it is called like being a part of Salt (this is a migration feature, not just generic execution implementation), while in your examples parameters are passed to somewhere else towards generic function target.

And even your example isn't the best one as it would be better to incorporate there the model I propose. Alongside of the weird path salt.runners.salt you have a) cmd and b) execute functions with the only difference that one should be remembered as for local stuff and the other one is exactly the same as previous, except works only on remote. It would be much cleaner to have "area.what.where <how>" where the what.where would replace what and anothersimilarwhat. E.g.:

salt.runners.execute.local <function> <parameters>
salt.runners.execute.remote <target> <function> <parameters>

And yes, unfortunately we have module.run instead of module.execute to the story above... 😞

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Sep 19, 2017

@thatch45 thoughts?

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Sep 19, 2017

@s0undt3ch do you have any ideas why tests on Ubuntu 16 fails that way? I am trying with PyTest and "traditional" tests in different directions and they are passing...

This is what happens on the Ubuntu 16:

Traceback (most recent call last):
  File "/testing/tests/unit/modules/test_ansiblegate.py", line 106, in test_resolver_module_loader_failure
    assert 'No module named ansible.modules.{0}'.format(mod) in str(import_error.value)
AssertionError

Different exception content?.. Maybe it will go away if I try now to see if the exception is raised at all. But this will kind of make the test a bit "dull" (too generic). Still I don't know how this env will react, since I am "poking the sky", in a hope it won't fail.

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Sep 20, 2017

Go Go Jenkins!

@cachedout cachedout requested a review from thatch45 Sep 20, 2017

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2017

@isbm Right now @thatch45 is out of the office but I will get him to weigh in here as soon as possible. Thanks for your patience.

if args:
kwargs['_raw_params'] = ' '.join(args)
js_args = '{{"ANSIBLE_MODULE_ARGS": {args}}}'.format(args=json.dumps(kwargs))
js_out = subprocess.Popen(["echo", "{0}".format(js_args)], stdout=subprocess.PIPE)

This comment has been minimized.

Copy link
@cachedout

cachedout Sep 29, 2017

Collaborator

What do you think about using salt.utils.timed_subprocess here and in the line below so that we can bail out if we hang?

This comment has been minimized.

Copy link
@isbm

isbm Oct 4, 2017

Author Contributor

Good point. Let me check for it. Getting back here ASAP.

Set all Ansible modules callables
:return:
'''
def _mkf(cmd_name, doc):

This comment has been minimized.

Copy link
@cachedout

cachedout Sep 29, 2017

Collaborator

A more descriptive function name here would help make this more maintainable in the future. :]

This comment has been minimized.

Copy link
@isbm

isbm Oct 4, 2017

Author Contributor

OK

@isbm isbm force-pushed the isbm:isbm-ansible-module-dev branch from 6a1fd40 to 1de723b Oct 12, 2017

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Oct 12, 2017

@cachedout OK, this now should be good enough.

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Oct 12, 2017

@cachedout I've also added an option ansible_timeout that can be set in /etc/salt/minion defaulting to 20 minutes and a documentation. No idea why GitHub tells me this branch is out of date, I just rebased...

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Oct 26, 2017

@cachedout anything else I need to do here?

@isbm isbm force-pushed the isbm:isbm-ansible-module-dev branch from 57f4ad1 to 59261b6 Nov 10, 2017

@isbm

This comment has been minimized.

Copy link
Contributor Author

commented Nov 10, 2017

@cachedout @thatch45 is there any chances to get this done? It is blocking further development of e.g. conversion Ansible playbooks to SLS. 😢 ...

@rallytime rallytime requested a review from cachedout Nov 10, 2017

@isbm isbm force-pushed the isbm:isbm-ansible-module-dev branch from 59261b6 to 8873a92 Nov 10, 2017

@cachedout cachedout merged commit 3af0031 into saltstack:develop Nov 13, 2017

6 of 9 checks passed

GPG All commits must have a verified GPG signature
Details
default Build finished.
Details
jenkins/PR/salt-pr-linode-ubuntu16-py3 Pull Requests » Salt PR - Linode Ubuntu16.04 - PY3 #3483 — FAILURE
Details
WIP ready for review
Details
codeclimate All good!
Details
jenkins/PR/salt-pr-clone Pull Requests » Salt PR - Clone #19056 — SUCCESS
Details
jenkins/PR/salt-pr-docs-n Pull Requests » Salt PR - Docs #11724 — SUCCESS
Details
jenkins/PR/salt-pr-linode-ubuntu14-n Pull Requests » Salt PR - Linode Ubuntu14.04 #16442 — SUCCESS
Details
jenkins/PR/salt-pr-lint-n Pull Requests » Salt PR - Code Lint #16291 — SUCCESS
Details

@isbm isbm deleted the isbm:isbm-ansible-module-dev branch Nov 14, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.