Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ansible 2.8 installs junk in sys.modules #590

Closed
dw opened this issue May 28, 2019 · 9 comments

Comments

Projects
None yet
2 participants
@dw
Copy link
Owner

commented May 28, 2019

The full traceback is:
Traceback (most recent call last):
  File "/home/dmw/src/mitogen/venv35/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 144, in run
    res = self._execute()
  File "/home/dmw/src/mitogen/venv35/lib/python3.5/site-packages/ansible/executor/task_executor.py", line 648, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/dmw/src/mitogen/ansible_mitogen/mixins.py", line 116, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/home/dmw/src/mitogen/venv35/lib/python3.5/site-packages/ansible/plugins/action/gather_facts.py", line 66, in run
    res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False)
  File "/home/dmw/src/mitogen/ansible_mitogen/mixins.py", line 367, in _execute_module
    timeout_secs=self.get_task_timeout_secs(),
  File "/home/dmw/src/mitogen/ansible_mitogen/planner.py", line 496, in invoke
    kwargs=planner.get_kwargs(),
  File "/home/dmw/src/mitogen/ansible_mitogen/connection.py", line 434, in call
    return self._rethrow(recv)
  File "/home/dmw/src/mitogen/ansible_mitogen/connection.py", line 420, in _rethrow
    return recv.get().unpickle()
  File "/home/dmw/src/mitogen/mitogen/core.py", line 835, in unpickle
    raise obj
mitogen.core.CallError: exceptions.ImportError: No module named _distro
  File "<stdin>", line 3108, in _dispatch_one
  File "master:/home/dmw/src/mitogen/ansible_mitogen/target.py", line 422, in run_module
    return impl.run()
  File "master:/home/dmw/src/mitogen/ansible_mitogen/runner.py", line 378, in run
    self.setup()
  File "master:/home/dmw/src/mitogen/ansible_mitogen/runner.py", line 786, in setup
    self._setup_imports()
  File "master:/home/dmw/src/mitogen/ansible_mitogen/runner.py", line 764, in _setup_imports
    mitogen.core.import_module(fullname)
  File "<stdin>", line 596, in import_module

Due to ansible.module_utils.distro which replaces itself in sys.modules with ansible.module_utils.distro._distro on 2.x or distro on 3.x.

@dw dw changed the title Ansible 2.8 installs junk in sys.modules on Python 3 Ansible 2.8 installs junk in sys.modules May 28, 2019

dw added a commit that referenced this issue May 30, 2019

dw added a commit that referenced this issue May 30, 2019

Merge remote-tracking branch 'origin/azure-280-35'
* origin/azure-280-35:
  issue #590: include nasty workaround for sys.modules junk
  azure: try enabling Ansible 2.8/Py3.5 job
@opoplawski

This comment has been minimized.

Copy link

commented May 30, 2019

I'm still seeing problems:

The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 144, in run
    res = self._execute()
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 648, in _execute
    result = self._handler.run(task_vars=variables)
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/mixins.py", line 116, in run                                                                                                 
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/lib/python2.7/site-packages/ansible/plugins/action/command.py", line 24, in run
    results = merge_hash(results, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/mixins.py", line 354, in _execute_module                                                                                     
    self._temp_file_gibberish(module_args, wrap_async)
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/mixins.py", line 333, in _temp_file_gibberish                                                                                
    self._connection.get_good_temp_dir()
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/connection.py", line 723, in get_good_temp_dir                                                                               
    self._connect()
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/connection.py", line 742, in _connect                                                                                        
    self._connect_stack(stack)
  File "/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/connection.py", line 696, in _connect_stack                                                                                  
    stack=mitogen.utils.cast(list(stack)),
  File "/export/home/orion-admin/ansible-boulder/mitogen/mitogen/core.py", line 1859, in call_service                                                                                                 
    return recv.get().unpickle()
  File "/export/home/orion-admin/ansible-boulder/mitogen/mitogen/core.py", line 835, in unpickle
    raise obj
CallError: exceptions.ImportError: The Mitogen master process was unable to serve u'ansible.module_utils.distro._distro'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.                                          
  File "<stdin>", line 3099, in _dispatch_one
  File "<stdin>", line 3086, in _parse_request
  File "<stdin>", line 596, in import_module
  File "<stdin>", line 1357, in load_module
  File "<string>", line 1, in <module>
  File "master:/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/target.py", line 85, in <module>                                                                                      
    import ansible_mitogen.runner
  File "<stdin>", line 1357, in load_module
  File "<string>", line 1, in <module>
  File "master:/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/runner.py", line 87, in <module>                                                                                      
    import ansible.module_utils.basic
  File "<stdin>", line 1357, in load_module
  File "<string>", line 1, in <module>
  File "master:/usr/lib/python2.7/site-packages/ansible/module_utils/basic.py", line 147, in <module>                                                                                                 
    from ansible.module_utils.common.sys_info import (
  File "<stdin>", line 1357, in load_module
  File "<string>", line 1, in <module>
  File "master:/usr/lib/python2.7/site-packages/ansible/module_utils/common/sys_info.py", line 11, in <module>                                                                                        
    from ansible.module_utils import distro
  File "<stdin>", line 1357, in load_module
  File "<string>", line 1, in <module>
  File "master:/usr/lib/python2.7/site-packages/ansible/module_utils/distro/__init__.py", line 46, in <module>                                                                                        
    from ansible.module_utils.distro import _distro as distro
  File "<stdin>", line 1330, in load_module
[mux  21079] 10:50:33.818375 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer().find_module('ansible.module_"..64), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                          
[mux  21079] 10:50:33.818614 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer().find_module('ansible.module_utils.distro')                                                                    
[mux  21079] 10:50:33.818782 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer.load_module(u'ansible.module_u"..63), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                          
[mux  21079] 10:50:33.818958 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer.load_module(u'ansible.module_utils.distro')                                                                     
[mux  21079] 10:50:33.819113 D mitogen.io: Broker(0x7fab76325890)._loop_once(None, EpollPoller(0x7fab76325990))                                                                                       
[mux  21079] 10:50:33.819273 D mitogen.io: EpollPoller(0x7fab76325990).poll(None)
[mux  21079] 10:50:33.821325 D mitogen.io: EpollPoller(0x7fab76325990): POLLIN: 79
[mux  21079] 10:50:33.821537 D mitogen.io: mitogen.ssh.Stream('ssh.kolea.cora.nwra.com').on_receive()                                                                                                 
[mux  21079] 10:50:33.821723 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer().find_module('distro')"..43), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                                 
[mux  21079] 10:50:33.821908 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer().find_module('distro')                                                                                         
[mux  21079] 10:50:33.822075 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer.load_module(u'distro')"..42), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                                  
[mux  21079] 10:50:33.822251 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer.load_module(u'distro')                                                                                          
[mux  21079] 10:50:33.822425 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer().find_module('ansible.module_"..72), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                          
[mux  21079] 10:50:33.822608 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer().find_module('ansible.module_utils.distro._distro')                                                            
[mux  21079] 10:50:33.822774 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer.load_module(u'ansible.module_u"..71), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                          
[mux  21079] 10:50:33.822958 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer.load_module(u'ansible.module_utils.distro._distro')                                                             
[mux  21079] 10:50:33.823126 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00_request_module(u'ansible.module_utils."..79), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                          
[mux  21079] 10:50:33.823302 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: _request_module(u'ansible.module_utils.distro._distro'): new request                                                     
[mux  21079] 10:50:33.823502 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 100, 0, 'ansible.module_utils.distro._distro'..35), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                                               
[mux  21079] 10:50:33.823689 D mitogen: ssh.kolea.cora.nwra.com requested module ansible.module_utils.distro._distro                                                                                  
[mux  21079] 10:50:33.823920 D mitogen.io: ModuleFinder()._get_module_via_pkgutil(u'ansible.module_utils.distro._distro') -> None                                                                     
[mux  21079] 10:50:33.824076 D mitogen: _get_module_via_sys_modules(u'ansible.module_utils.distro._distro') -> None                                                                                   
[mux  21079] 10:50:33.824228 D mitogen: sys.modules[u'ansible.module_utils.distro._distro'] absent or not a regular module                                                                            
[mux  21079] 10:50:33.824376 D mitogen: get_module_source(u'ansible.module_utils.distro._distro'): cannot find source                                                                                 
[mux  21079] 10:50:33.824549 D mitogen: _build_tuple(u'ansible.module_utils.distro._distro'): could not locate source                                                                                 
[mux  21079] 10:50:33.824714 D mitogen: ssh.kolea.cora.nwra.com: sending module ansible.module_utils.distro._distro (0.05 KiB)                                                                        
[mux  21079] 10:50:33.824884 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(2, 0, 0, 107, None, '\x80\x02(X#\x00\x00\x00ansible.module_utils.distro._distroq\x01NNN)t'..53), None) 
[mux  21079] 10:50:33.825054 D mitogen.io: mitogen.ssh.Stream('ssh.kolea.cora.nwra.com')._send(Message(2, 0, 0, 107, None, '\x80\x02(X#\x00\x00\x00ansible.module_utils.distro._distroq\x01NNN)t'..53))                                                                                                  
[mux  21079] 10:50:33.825273 D mitogen.io: Broker(0x7fab76325890)._loop_once(None, EpollPoller(0x7fab76325990))                                                                                       
[mux  21079] 10:50:33.825494 D mitogen.io: EpollPoller(0x7fab76325990).poll(None)
[mux  21079] 10:50:33.826835 D mitogen.io: EpollPoller(0x7fab76325990): POLLIN: 79
[mux  21079] 10:50:33.827049 D mitogen.io: mitogen.ssh.Stream('ssh.kolea.cora.nwra.com').on_receive()                                                                                                 
[mux  21079] 10:50:33.827240 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, "mitogen\x0010\x00Importer._on_load_module(u'ansible.modu"..75), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                          
[mux  21079] 10:50:33.827445 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: Importer._on_load_module(u'ansible.module_utils.distro._distro')                                                         
[mux  21079] 10:50:33.827608 D mitogen.io: Broker(0x7fab76325890)._loop_once(None, EpollPoller(0x7fab76325990))                                                                                       
[mux  21079] 10:50:33.827760 D mitogen.io: EpollPoller(0x7fab76325990).poll(None)
[mux  21079] 10:50:33.837565 D mitogen.io: EpollPoller(0x7fab76325990): POLLIN: 79
[mux  21079] 10:50:33.837785 D mitogen.io: mitogen.ssh.Stream('ssh.kolea.cora.nwra.com').on_receive()                                                                                                 
[mux  21079] 10:50:33.837987 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 102, 0, 'mitogen\x0010\x00_dispatch_calls: Message(2, 0, 0, 101, '..1813), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                                        
[mux  21079] 10:50:33.838171 D mitogen.ctx.ssh.kolea.cora.nwra.com: mitogen: _dispatch_calls: Message(2, 0, 0, 101, 1000, '\x80\x02(NX\x16\x00\x00\x00ansible_mitogen.targetNX\n\x00\x00\x00init_child)cm'..164) -> CallError(u'exceptions.ImportError: The Mitogen master process was unable to serve u\'ansible.module_utils.distro._distro\'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.\n  File "<stdin>", line 3099, in _dispatch_one\n  File "<stdin>", line 3086, in _parse_request\n  File "<stdin>", line 596, in import_module\n  File "<stdin>", line 1357, in load_module\n  File "<string>", line 1, in <module>\n  File "master:/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/target.py", line 85, in <module>\n    import ansible_mitogen.runner\n  File "<stdin>", line 1357, in load_module\n  File "<string>", line 1, in <module>\n  File "master:/export/home/orion-admin/ansible-boulder/mitogen/ansible_mitogen/runner.py", line 87, in <module>\n    import ansible.module_utils.basic\n  File "<stdin>", line 1357, in load_module\n  File "<string>", line 1, in <module>\n  File "master:/usr/lib/python2.7/site-packages/ansible/module_utils/basic.py", line 147, in <module>\n    from ansible.module_utils.common.sys_info import (\n  File "<stdin>", line 1357, in load_module\n  File "<string>", line 1, in <module>\n  File "master:/usr/lib/python2.7/site-packages/ansible/module_utils/common/sys_info.py", line 11, in <module>\n    from ansible.module_utils import distro\n  File "<stdin>", line 1357, in load_module\n  File "<string>", line 1, in <module>\n  File "master:/usr/lib/python2.7/site-packages/ansible/module_utils/distro/__init__.py", line 46, in <module>\n    from ansible.module_utils.distro import _distro as distro\n  File "<stdin>", line 1330, in load_module\n',)  
[mux  21079] 10:50:33.838373 D mitogen.io: Router(Broker(0x7fab76325890))._async_route(Message(0, 2, 2, 1000, 0, '\x80\x02cmitogen.core\n_unpickle_call_error\nq\x01XY\x06\x00\x00except'..1676), mitogen.ssh.Stream('ssh.kolea.cora.nwra.com'))                                                         
@opoplawski

This comment has been minimized.

Copy link

commented May 30, 2019

# rpm -ql ansible | grep distro
/usr/lib/python2.7/site-packages/ansible/module_utils/distro
/usr/lib/python2.7/site-packages/ansible/module_utils/distro/__init__.py
/usr/lib/python2.7/site-packages/ansible/module_utils/distro/__init__.pyc
/usr/lib/python2.7/site-packages/ansible/module_utils/distro/__init__.pyo
/usr/lib/python2.7/site-packages/ansible/module_utils/distro/_distro.py
/usr/lib/python2.7/site-packages/ansible/module_utils/distro/_distro.pyc
/usr/lib/python2.7/site-packages/ansible/module_utils/distro/_distro.pyo
@opoplawski

This comment has been minimized.

Copy link

commented May 30, 2019

Perhaps this is because I have python2-distro on the controller but not the target?

@opoplawski

This comment has been minimized.

Copy link

commented May 30, 2019

yeah, it works if I remove python2-distro from the master.

@dw

This comment has been minimized.

Copy link
Owner Author

commented Jun 2, 2019

Possible configurations:

  • A) 2x controller, no python-distro, sys.modules[...distro] replaced by distro._distro

  • B) 2x controller, python-distro, sys.modules[...distro] is replaced by "distro" from 3rd party

  • C) 3x controller, sys.modules[...distro] is replaced by "distro" from standard library

  • A1) 2x target, no python-distro, controller A) must serve unmutilated original distro/init.py and distro._distro

  • A2) 2x target, python-distro, controller A) must serve distro/init.py only

  • A3) 3x target, controller A) must serve distro/init.py only

  • B1) 2x target, no python-distro, controller B) must serve ??. I think same as A1)

  • B2) 2x target, python-distro, controller B) must serve distro/init.py only

  • B3) 3x target, controller B) must serve distro/init.py only

  • C1) 2x target, no python-distro, controller C) must serve distro/init.py and distro._distro

  • C2) 2x target, python-distro, controller C) must serve distro/init.py only

  • C3) 3x target, controller C) must serve distro/init.py only


Since this is damage done on the controller, the module source override hook isn't useful.

A better option than trying to hack around this specific problem might be to teach the module importer about the general case of errant software that corrupts sys.modules in a similar manner. That requires verifying __name__ of a module value found in sys.modules against the key used to look it up, and falling back to using the filesystem rather than memory in this case.

@dw

This comment has been minimized.

Copy link
Owner Author

commented Jun 2, 2019

Replacing distro/__init__.py with below is sufficient to get everything working again, I wonder how receptive upstream would be to such a patch

from __future__ import absolute_import
try:
    from distro import *
except ImportError:
    from ansible.module_utils.distro._distro import *
@dw

This comment has been minimized.

Copy link
Owner Author

commented Jun 2, 2019

This is further complicated by the fact 'distro' is magically switched from a package to a module. Given 2x controller -> 3x target, the 2x controller will send 'distro._distro' as part of the set of module preloads, but after preloading 'distro' during runner initialization, on 3.x the package is replaced with the stdlib 'distro' before the importer ever runs again to make use of the package metadata that is about to be discarded.

So on 3.x, by the time 'distro._distro' is preloaded, the interpreter is incapable of importing that module, because the required state has been trampled over. Looks like this needs a permanent workaround in in the runner for this single module. Such a trainwreck

dw added a commit that referenced this issue Jun 2, 2019

issue #590: refactor ModuleFinder and teach it a new special case.
Now it's possible to find both packages and modules when the
sys.modules[...] state for the package/module is junk. Previously only
modules were possible.

This also refactors things to make writing better tests for all these
cases much simpler.

dw added a commit that referenced this issue Jun 2, 2019

dw added a commit that referenced this issue Jun 2, 2019

dw added a commit that referenced this issue Jun 2, 2019

dw added a commit that referenced this issue Jun 2, 2019

dw added a commit that referenced this issue Jun 2, 2019

dw added a commit that referenced this issue Jun 2, 2019

Merge remote-tracking branch 'origin/issue590'
* origin/issue590:
  issue #590: disable distro test on vanilla
  ci: Ansible 2.8 jobs aren't running against all host types.
  tests: Py3.x fix.
  issue #590: fix test for <2.8 Ansibles.
  tests: Py3.x fix.
  issue #590: actually run Ansible test.
  tests: Py3.x fix.
  master: sysconfig did not exist until 2.7.
  tests: rearrange test modules again, they're used in multiple places
  module_finder_test: mask one more difference between unit2 vs. direct start
  master: fix _is_stdlib_path() failure on Ubuntu.
  issue #590: add dummy package for new test.
  issue #590: add FinderMethod docstrings.
  issue #590: update comment to indicate the hack is permanent
  issue #590: move example modules to module_finder/, fix/add tests
  issue #590: refactor ModuleFinder and teach it a new special case.
  issue #590: Ansible test for module_utils.distro use.
  issue #590: teach importer to handle self-replacing modules
@dw

This comment has been minimized.

Copy link
Owner Author

commented Jun 2, 2019

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://www.freelists.org/list/mitogen-announce

Thanks for reporting this!

@dw dw closed this Jun 2, 2019

@opoplawski

This comment has been minimized.

Copy link

commented Jun 3, 2019

So far, so good. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.