Allow dynamic imports in plugins.

[flungo]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

module_utils

ANSIBLE VERSION

ansible 2.3.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.13 (default, Jul 21 2017, 03:24:34) [GCC 7.1.1 20170630]

CONFIGURATION

No specific configuration

OS / ENVIRONMENT

N/A

SUMMARY

It does not seem that there is at present a way to have common code between plugins provided adjacent to a playbook or in a role. This can result in plugin files having a lot of common code which has to be maintained in multiple places. I think it would be beneficial for there to be a way of sharing code between plugins. This is possible (and common) within the packaged plugins and will help with the development of new plugins.

For modules, a modules_utils directory can be included adjacent to the playbook or inside the role which can then be accessed by custom modules to provide common functionality. This is extremely useful but does not work with plugins. Whether it is by making modules_utils inside action plugins or by creating some other directory for common code, I would like to see the

If I am not mistaken, plugins all run locally and modules run on the remote host. As such I assume that all module_utils are sent to the remote host. Given this, it may be beneficial to have plugin_utils which are only available to plugins and are hence not sent to remote hosts. This is not to say that module_utils shouldn't also be available to plugins to avoid the need for the same code in plugin_utils and module_utils.

STEPS TO REPRODUCE

I have created a small project which shows that the files under module_utils can be used

To run this sample, use ansible -vvv playbook.yml.

EXPECTED RESULTS

I would like the common code to be imported and for the module to run successfully with the following output for the second task:

TASK [Get the time using the plugin] ********************************************************************
task path: /home/flungo/developer/ansible/module_util_test/playbook.yml:5
ok: [localhost] => {
    "changed": false, 
    "datetime": "2017-08-29 14:58:19.903485"
}

ACTUAL RESULTS

The import fails and so the module cannot be loaded:

ERROR! Unexpected Exception: No module named datetime_helper
the full traceback was:

Traceback (most recent call last):
  File "/usr/bin/ansible-playbook", line 109, in <module>
    exit_code = cli.run()
  File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 154, in run
    results = pbex.run()
  File "/usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py", line 153, in run
    result = self._tqm.run(play=play)
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 288, in run
    play_return = strategy.run(iterator, play_context)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/strategy/linear.py", line 203, in run
    action = action_loader.get(task.action, class_only=True)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 358, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/__init__.py", line 343, in _load_module_source
    module = imp.load_source(full_name, path, module_file)
  File "/home/flungo/developer/ansible/module_util_test/action_plugins/datetime_plugin.py", line 2, in <module>
    from ansible.module_utils.datetime_helper import get_datetime
ImportError: No module named datetime_helper

[ansibot]

Files identified in the description:

• lib/ansible/module_utils

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[baughj]

👍 - this would be very helpful for making custom lookup code reusable / modular.

[abadger]

Unlike module_utils, this doesn't seem very useful except as a matter of symmetry. You can just install into a local pythonpath for controller code. To keep everything contained in the role, you can do something like include a python wheel or zipfile of the code you need to install in the roles files and then use the pip module to install that on the local system as one of the role's first tasks.

[baughj]

I won't speak to @flungo 's original use case / reason for opening this ticket, but for me, it would be useful to keep everything in one git repository without the need for shims or some kind of notification/documentation that PYTHONPATH must be set to a local path. For my use case, we would want users to be able to simply git checkout and use Ansible without any other interventions. Symmetry in my mind makes this less astonishing, too, which is always good (since there is a reasonable expectation, at least in my mind, that plugins would work in a somewhat similar fashion as modules).

[abadger]

I think that the example I sketched above satisfies all of your needs.

[abadger]

Here's a sample of the example to make clear what I mean: https://github.com/abadger/sample-28770

[bcavns01]

Here's a sample of the example to make clear what I mean: https://github.com/abadger/sample-28770

Your sample is definitely clear, but having to create the scaffolding to create a pip package and then install/maintain it is a lot of paperwork and, more importantly, a lot of added complexity, particularly if you have multiple small libraries. Aside from symmetry, having this as a native Ansible feature would lead to simpler playbooks and cleaner, more self-contained Ansible repos.

I see that Ansible is pretty much just doing some vanilla imp calls to load in the filter plugins, and it looks like a few minor additions near the top of the all method in ansible/plugins/loader.py would produce this feature.

The feature is so close to already existing it can almost be achieved with just some "hackery":
#27748 (comment)

[sumkincpp]

I've took sometime to check loader implementation and got some ideas here.

I believe that after #37748 we are unable to import filter plugins via imp.load_source(..) because they are imported with package_names consisting of full path to the file.
For example, ipaddr is imported as

ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/ipaddr

So, we are obviously can't reference this module it from other filter/module.

And this is true for filters,tests,callbacks:

 [WARNING]: ansible.plugins.test./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/test/core
 [WARNING]: ansible.plugins.test./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/test/files
 [WARNING]: ansible.plugins.test./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/test/mathstuff
 [WARNING]: ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/core
 [WARNING]: ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/ipaddr
 [WARNING]: ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/json_query
 [WARNING]: ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/mathstuff
 [WARNING]: ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/network
 [WARNING]: ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/urlsplit

I have a suggestion to revert module package name to old behaviour, so that package_name will be correct(e.g. ansible.plugins.filter.ipaddr), but if we are still hitting duplicate package names, just load it with a new naming(e.g. ansible.plugins.filter./opt/awx/awx-app/venv/lib/python2.7/site-packages/ansible/plugins/filter/ipaddr) and display warning to user.

@abadger, what do you think?

[cognifloyd]

I would also like a way to share code between my inventory, action, and lookup plugins.
I'm indifferent between module_utils or something else.

@abadger Your example is interesting, I don't want my playbooks to do local pip installs because ansible is running in one of various virtualenvs on the controller, so assuming you can access --user is not always true. I'm currently installing a python module in those virtualenvs with some shared code, but I don't want to maintain the machinery to keep those virtualenvs up to date with the playbooks repo as I update it. In several cases, the virtualenv belongs to some other program, and that program doesn't provide any hooks to allow for installing additional packages beyond its own requirements.txt, so I don't like modifying the virtualenv in case that program modifies or replaces the virtualenv (eg during an update).

[cognifloyd]

I'm on ansible 2.7, but I also considered using content collections with ansible 2.8 / mazer.
However, it seems that there is not a way to share code in a collection between plugins in that collection.

[cognifloyd]

@nitzmahone (responding to your #48844 (comment) ) Is there a way to share code between controller-side plugins with collections? How does the collections namespacing work in Ansible 2.8?

There's a significant change planned for pluginloader in 2.8 that will allow for a more controlled form of this behavior (which this PR would conflict with). ... we can revisit once 2.8 has shipped if you're still not able to get what you need from the new stuff. ... basically we're adding namespacing capability to content that's been "installed" (eg from galaxy) where you can safely reference other installed content by namespace (rather than relative file paths).

edit: to add this comment:
It looks like the ansible_collections.<namespace>.<collection> implicit namespace package is explicitly importable from within modules (as expected).
The plugins themselves are importable due to how the CollectionLoader runs a types.ModuleType through exec, but it's not clear (from reading the code) if relative imports or other from ansible_collections.<namespace>.<collection>.plugins.module_utils import ... or from ansible_collections.<namespace>.<collection>.plugins.<plugin_type>.<plugin> import ... will work within those plugins.
In any case, none of the collections tests have imports in a controller-side plugin (inventory, filter, test, lookup, action, etc) that pull from somewhere else in the collection.

[bcavns01]

Haven't looked into namespaces in 2.8, but an additional work around in 2.7 through 2.8, better than one I posted in an earlier issue, is to use inventory_plugins as the location for your shared code. This is still a hack, and it won't help sharing between inventory plugins (unless maybe you're careful with filenames in the same way as my previous workaround), but it works great if you are interested in sharing code between filter plugins.

As an example, if your top-level inventory plugins directory is inventory_plugins, you can create a module (mymodule.py) in it like this:

__metaclass__ = type

from ansible.plugins.inventory import BaseInventoryPlugin
from ansible import errors

class InventoryModule(BaseInventoryPlugin):
    NAME = 'mymodule'

    def parse(self, inventory, loader, path, cache=True):
        return 

def myfunction(params):
	return("DID SOME STUFF!")

And then you can import in your filter plugins like this:
from ansible.plugins.inventory.mymodule import myfunction

[nitzmahone]

@cognifloyd Because collections-hosted controller plugins are loaded and instantiated by a PEP302 loader (bypassing a lot of the "fun" in PluginLoader), and all collections are valid Python packages/namespaces by definition, you can freely reference types within (and even across) collections (eg, from ansible_collections.myns.mycoll.plugins.inventory import a_thing). I haven't done much testing with relative imports in collections, but at one point that did work correctly for code that exclusively runs in the controller. Modules and module_utils do not currently support relative imports for code that needs to be shipped to the target (the import walker will ignore them)- we've discussed it as a possibility (that would certainly make imports in collections more developer-friendly), but it's not a roadmapped feature right now.

[sgburtsev]

As problem is still actual, this is my approach.
I created a module in module_utils directory as usual and placed it in the config:

module_utils = module_utils

Then I tried to load this module into a custom Action plugin with module_utils_loader:

class ActionModule(ActionBase):
    def run(self, tmp=None, task_vars=None):
        module = self._shared_loader_obj.module_utils_loader.get('custom', class_only=True)

But it was unsuccessful:

The full traceback is:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/ansible/executor/task_executor.py", line 146, in run
    res = self._execute()
  File "/usr/local/lib/python3.7/site-packages/ansible/executor/task_executor.py", line 645, in _execute
    result = self._handler.run(task_vars=variables)
  File "/Users/s.burtsev/work/ansible/plugins/action/generate_dc.py", line 36, in run
    obj = self._shared_loader_obj.module_utils_loader.get('generate_dc')
  File "/usr/local/lib/python3.7/site-packages/ansible/plugins/loader.py", line 561, in get
    obj = getattr(self._module_cache[path], self.class_name)
AttributeError: module 'ansible.module_utils.custom' has no attribute ''
fatal: [localhost]: FAILED! =>
  msg: Unexpected failure during module execution.
  stdout: ''

So I had to add a little hack into my module:

def __getattr__(name):
    import sys
    if name == '':
        return sys.modules[__name__]

Thus you could use your module_utils in modules and in plugins as well.

[nitzmahone]

Not sure what you're trying to accomplish here, but this is almost certainly not the right way to do it. You can do this trivially with a playbook-adjacent collection in 2.8+ with no such hackery required.

[nitzmahone]

I'm actually going to close this issue, since collections satisfies the original request...