[ansible] Imports from module_utils fail

[ti-mo]

Hi, thanks for the awesome project, looks very promising so far! We're running into some import issues with some custom modules we have. As a way of sharing libraries between modules (and setting up their remote dependencies if required), we 'package' some module_utils in a separate role. This plays more or less nicely with Ansible's upstream module importer.

In the example below, the file layout is as follows:

roles/
  |- calico-modules/
    |- meta/main.yml -> depends on 'libraries' role
    |- action_plugins/calicoctl.py (works and imports fine)
    |- library/calicoctl.py (called by the action plugin)
      > from ansible.module_utils.exec_utils import find_executable
  |- libraries/module_utils/exec_utils.py
    > 1: import os

However, any imports in those module_utils files trip up Mitogen. I've been able to get rid of most of them, but replacing os is going to be difficult. Does this scenario fall under the '“Module Replacer” style modules are not supported.' clause, and if so, what would be the recommended way of structuring this?

The full traceback is:
Traceback (most recent call last):
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 138, in run
    res = self._execute()
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 554, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 115, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/home/ansible/ansible-common/roles/calico-modules/action_plugins/calicoctl.py", line 56, in run
    task_vars=task_vars, tmp=tmp)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 337, in _execute_module
    timeout_secs=self.get_task_timeout_secs(),
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/planner.py", line 477, in invoke
    response = _invoke_forked_task(invocation, planner)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/planner.py", line 442, in _invoke_forked_task
    kwargs=planner.get_kwargs(),
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/mitogen/parent.py", line 997, in call
    return receiver.get().unpickle(throw_dead=False)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/mitogen/core.py", line 487, in unpickle
    raise obj
CallError: exceptions.ValueError: __package__ set to non-string
  File "<stdin>", line 2036, in _dispatch_calls
  File "<stdin>", line 2028, in _dispatch_one
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/target.py", line 263, in run_module
    return impl.run()
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 225, in run
    self.setup()
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 535, in setup
    self._setup_imports()
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 519, in _setup_imports
    mitogen.core.import_module(fullname)
  File "<stdin>", line 367, in import_module
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 270, in load_module
    exec(code, mod.__dict__)
  File "/home/ansible/ansible-common/roles/libraries/module_utils/exec_utils.py", line 1, in <module>

The first line of exec_utils.py is import os.

I've tried both py2 and py3 on the controller, and the results are the same. (nice!) The remotes are CentOS7 machines running py2.

Is this a scenario Mitogen could support?

[dw]

80% chance this is some easy fallout from the Python 3 work. Leave this with me a few days, there are some related tests checked in that don't run yet, so I'll fix that at the same time. Thanks for the report!

[dw]

Hi there,

Can you please retry with http://github.com/dw/mitogen/archive/dmw.zip and let me know if the problem is gone? I need to rework the module_utils tests more significantly, because Ansible's semantics when overlapping module names exists is pretty unfortunate

[ti-mo]

@dw works perfectly, thank you! (tested with py2 and py3 runners)

[dw]

This will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!

[dw]

Oh heck, sorry, I shouldn't have pointed you at that branch. The tip commit had the per-CPU multiplexer patch on top. If you re-pull (assuming you're using it temporarily) that patch is gone. It may be fine (it might even be great!), but best not use it yet.