New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ansible] Imports from module_utils fail #324

Closed
ti-mo opened this Issue Jul 26, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@ti-mo

ti-mo commented Jul 26, 2018

Hi, thanks for the awesome project, looks very promising so far! We're running into some import issues with some custom modules we have. As a way of sharing libraries between modules (and setting up their remote dependencies if required), we 'package' some module_utils in a separate role. This plays more or less nicely with Ansible's upstream module importer.

In the example below, the file layout is as follows:

roles/
  |- calico-modules/
    |- meta/main.yml -> depends on 'libraries' role
    |- action_plugins/calicoctl.py (works and imports fine)
    |- library/calicoctl.py (called by the action plugin)
      > from ansible.module_utils.exec_utils import find_executable
  |- libraries/module_utils/exec_utils.py
    > 1: import os

However, any imports in those module_utils files trip up Mitogen. I've been able to get rid of most of them, but replacing os is going to be difficult. Does this scenario fall under the '“Module Replacer” style modules are not supported.' clause, and if so, what would be the recommended way of structuring this?

The full traceback is:
Traceback (most recent call last):
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 138, in run
    res = self._execute()
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 554, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 115, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/home/ansible/ansible-common/roles/calico-modules/action_plugins/calicoctl.py", line 56, in run
    task_vars=task_vars, tmp=tmp)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/mixins.py", line 337, in _execute_module
    timeout_secs=self.get_task_timeout_secs(),
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/planner.py", line 477, in invoke
    response = _invoke_forked_task(invocation, planner)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/planner.py", line 442, in _invoke_forked_task
    kwargs=planner.get_kwargs(),
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/mitogen/parent.py", line 997, in call
    return receiver.get().unpickle(throw_dead=False)
  File "/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/mitogen/core.py", line 487, in unpickle
    raise obj
CallError: exceptions.ValueError: __package__ set to non-string
  File "<stdin>", line 2036, in _dispatch_calls
  File "<stdin>", line 2028, in _dispatch_one
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/target.py", line 263, in run_module
    return impl.run()
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 225, in run
    self.setup()
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 535, in setup
    self._setup_imports()
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 519, in _setup_imports
    mitogen.core.import_module(fullname)
  File "<stdin>", line 367, in import_module
  File "master:/home/ansible/.virtualenvs/ansible-27/lib/python2.7/site-packages/ansible_mitogen/runner.py", line 270, in load_module
    exec(code, mod.__dict__)
  File "/home/ansible/ansible-common/roles/libraries/module_utils/exec_utils.py", line 1, in <module>

The first line of exec_utils.py is import os.

I've tried both py2 and py3 on the controller, and the results are the same. (nice!) The remotes are CentOS7 machines running py2.

Is this a scenario Mitogen could support?

@dw

This comment has been minimized.

Owner

dw commented Jul 26, 2018

80% chance this is some easy fallout from the Python 3 work. Leave this with me a few days, there are some related tests checked in that don't run yet, so I'll fix that at the same time. Thanks for the report!

@dw dw added bug ansible labels Jul 28, 2018

dw added a commit that referenced this issue Jul 28, 2018

issue #324: fix Python 3 fallout for custom module_utils.
Also enable at last one of its tests.
@dw

This comment has been minimized.

Owner

dw commented Jul 28, 2018

Hi there,

Can you please retry with http://github.com/dw/mitogen/archive/dmw.zip and let me know if the problem is gone? I need to rework the module_utils tests more significantly, because Ansible's semantics when overlapping module names exists is pretty unfortunate

@dw dw added the NeedsTest label Jul 28, 2018

@ti-mo

This comment has been minimized.

ti-mo commented Jul 30, 2018

@dw works perfectly, thank you! (tested with py2 and py3 runners)

@dw

This comment has been minimized.

Owner

dw commented Jul 30, 2018

This will make it into the next release. To be updated when a new release is made, subscribe to https://www.freelists.org/list/mitogen-announce

Thanks for reporting this!

@dw dw closed this Jul 30, 2018

@dw

This comment has been minimized.

Owner

dw commented Jul 30, 2018

Oh heck, sorry, I shouldn't have pointed you at that branch. The tip commit had the per-CPU multiplexer patch on top. If you re-pull (assuming you're using it temporarily) that patch is gone. It may be fine (it might even be great!), but best not use it yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment