multi-file custom modules (with ModuleReplacer), outside /ansible/module...

[sirkubax]

This change allows writing multi-file ansible modules (and store them outside ansible/module_utils/ directory - for easy ansible core upgrade).
The custom_modules can have external imports, that would be recognized by ModuleReplacer.

The idea is, that you can configure you custom modules directory (like other ansible vars):
EXAMPLE:

• via env: ANSIBLE_CUSTOM_MODULES
• (or ansible.cfg) custom_modules = some_path/lib/modules/

And place there Your files:

some_path/lib/
                   modules/
                            conf.py
                            tools.py
                            web-service.py
                            websrv -> web-service.py

Then just include 'ansible.module_custom' in Your custom module (mine -> websrv):

web-service.py:

from ansible.module_custom.tools import *
from ansible.module_custom.conf import *
import time
#and still You can import from ansible module_utils
#from ansible.module_utils.basic import *

#some code

and that would be merged by ModuleReplacer.

---

Info - in the ansible.cfg I have already 'library_path'

ansible.cfg
custom_modules = some_path/lib/modules/
library_path = some_path/lib/modules/

We could use the same variable, but it would be wise to have this configuration separated.

[srgvg]

+1

[bcoca]

the name custom_modules will confuse people, these are really module copied requirements or dependencies

I would also look at plugin loader to allow for local paths, the way other plugins do, this way a module that comes with a play or role can have their extra dependencies also provided vs needing to install them in a specific ansible location.

[bcoca]

this assumes that the direct import will be the only thing needed, what about dependencies of that library?

should we allow for a path or archive to copy?

[sirkubax]

the name custom_modules will confuse people, these are really module copied requirements or dependencies

Please suggest the name of Your choice.
What about custom_lib_module?

I would also look at plugin loader to allow for local paths, the way other plugins do, this way a module that comes with a play or role can have their extra dependencies also provided vs needing to install them in a specific ansible location.

I think there is a question: how deep should we include dependences into single merged_module_file. I would suggest, that the 'standard python packages' should be installed (or manually delivered if needed) into remote host environment.

this assumes that the direct import will be the only thing needed, what about dependencies of that library?

That's true, for now it does.
I did just extend the current ansible functionality (where modules are almost self-contained).

For now, I do assume, that the required custom modules (and their dependencies) would be provided in the ANSIBLE_CUSTOM_MODULES path.

For custom_lib_module depending on another custom_lib_module:
Let's make an example: a lib file (in example: tools.py) depends on the other lib file (conf.py), then if You would import both into Your module (websrv), the ModuleReplacer would merge content of all files (websrv, tools.py, conf.py) into one, and the dependence: tools.py -> conf.py would be resolved by this phenomena.

I do assume, that the other dependencies would be just a python packages - that should be installed on the remote hosts (this can be done by other play).

should we allow for a path or archive to copy?

I'm not sure if I do understand this part of the question.

---

IDEA
We could also think about using the eggs.
Provided we store them in customized location on local host, we have a few ways of importing an egg into our merged_module_file.
We can:

• append eggs into merged_module_file (and loading them on_the_fly)
• sending them to the remote host before running the module, and import their content

[juliedavila]

@sirkubax can you fix the merge conflicts when you get a chance?

[sivel]

I think ideally this functionality would be better implemented using plugin functionality, instead of a config.

[juliedavila]

@sivel agreed for a basic use case. what I feel this could solve is having the equivalent of an end-user managed equivalent of module_utils/$somelib so as to avoid duplication of python functions between their custom modules. There may be a better way, internally it's being discussed. Perhaps leverage zip imports, etc.

[sirkubax]

@defionscode
Sure, I can update the solution.
Should we cover current solution, or update the idea?

[amoralej]

Is there any other way to implement this use currently?, I'm hitting it for a custom module I'm writting. There should be a way to include snippets from files out of module_utils dir.

[juliedavila]

It's being worked on internally at the moment.

[alem0lars]

I'm having the same problem. Is this feature already integrated into the core?

[sirkubax]

Heh :)
Never thought it would evolve this way :)

I personally use for the moment a private fork of ansible repo with this changes.

For the future:
Extending the functionality with 'zip/egg import' would be great too.
@defionscode What is the internal status? Is there discussion we could take a part with?

[bcoca]

@sirkubax discussions are all public, either here on tickets or in mailing lists (ansible-devel) or irc (#ansible-devel)

currently this is waiting for the ziploader addition that we are targeting for 2.1

[jseadragon]

Now that Ansible 2.1 is out, is there anything blocking this from getting wrapped up? I'd love to see this pushed.

[linuxdynasty]

I actually would like this to happen as well for my PR #17039.
In order to get around this limitation I am doing import ansible.module_utils.ec2 as ec2, so that I can import my decorator.

[jdatx]

any update on this? I would also really like this feature

[bcoca]

so ziploader is already been out for 2.1 and this can be adapted to use that.
cc @abadger

[alem0lars]

Is this pull request solved in master branch?

[jseadragon]

Is there any traction on this, or on using ziploader to make this happen? It would be a really helpful feature for a couple libraries I'm writing that use shared code. If I can help, let me know.

If all that's needed is a rebase / conflict resolve, I could probably do it.

[sirkubax]

hm...
As an author of this idea, I might try to find some time to prepare a PR, if I'd gather more data:

@bcoca @defionscode - could you provide some summary on current project status?
Any working example on how you would like to include egg/zipfile/[flat python files?]

[BTW - I was using private fork of ansible 1.9.x , but I did recently upgrade to 2.x and the project structure has changed a lot]

[ansibot]

@sirkubax This PR was tested by travis-ci.org, which is no longer used. Please rebase your branch to trigger running of current tests.

click here for bot help

[abadger]

I thought @jimi-c had a WIP branch somewhere but all I can find is this: https://github.com/ansible/ansible/tree/module_utils_commmon_loading He definitely has something newer somewhere....

[jimi-c]

Yes, I have a different branch (which is not ready), but which supplants this one. As such, we'll close this now.

Thanks!

[sublee]

Is this issue fixed?

[tarasmatsyk]

Hi,
if anyone is still having an issue I have good news for you.
Ansible 2.3 realese introduced a feature to share code between custom modules the same way it works with module_utils.

Just place you shared code in module_utils folder and import it as you would import a default module
from ansible.module_utils.my_shared_code import MySharedCodeClient

There is a different option to use a ANSIBLE_MODULE_UTILS_PATH configuration but I have not figured out it yet :D

[sublee]

Awesome!

[alem0lars]

Yeah thanks

…

On May 25, 2017 13:19, "Heungsub Lee" ***@***.***> wrote: Awesome! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAUkPSlIRfCuAKLMojKJoglKSZInvH3Sks5r9WPfgaJpZM4DiBsG> .

[m4grio]

Solution mentioned by @tarasmatsyk is not working to me – maybe someone can shed some light?

I need to import a module called foo.py from an action plugin. I've put it into module_utils/foo.py.
In the plugin – that is stored at action_plugins/super_plugin.py I'm doing:

from ansible.module_utils.foo import FooClass

FooClass is in foo.py and I'm able to use it without Ansible, but when the plugin is called, I'm getting:

ERROR! Unexpected Exception: No module named foo

What am I doing wrong?

ansible --version
ansible 2.3.0.0
  config file = ./ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.5 (default, Aug  2 2016, 04:20:16) [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]

I'm asking for help first here rather than opening up an issue because I believe I'm doing something wrong (or rather stu**d).

[abadger]

Note: this is a user question. Better to take it to the mailing list or irc channel: http://docs.ansible.com/ansible/community.html#mailing-list-information

action plugins (and indeed any other plugins; only modules are different) have access to all of the python libraries installed on the controller. They do not have access to the module_utils that can be bundled as plugins as they are not installed. The module_utils plugin functionality only comes into play for modules (which are bundled up with their module_util dependencies and transferred over the wire to the remote machine).

For plugins, the libraries need to be installed locally on the controller instead.