Mitogen broken by distro 1.7.0

[stefanor]

From https://bugs.debian.org/1006330:

Something like issue #590 (see: #610, #622) seems to be back with distro 1.7.0. Presumably because of the change of module layout in python-distro/distro#315

With the Debian packaged ansible 2.10.8, mitogen 0.3.1, on Python 3.9.8. Upgrading from distro 1.6.0 to 1.7.0 on the target broke mitogen:

[mux  685121] 14:38:50.354759 D mitogen: PkgutilMethod(): loading 'ansible.module_utils.distro' using <_frozen_importlib_external.SourceFileLoader object at 0x7fa6329f7700> failed: loader for distro cannot handle ansible.module_utils.distro
[mux  685121] 14:38:50.354915 D mitogen: _get_module_via_sys_modules('ansible.module_utils.distro') -> <module 'distro' from '/usr/lib/python3/dist-packages/distro/__init__.py'>
[mux  685121] 14:38:50.355021 D mitogen: sys.modules['ansible.module_utils.distro'].__name__ is incorrect, assuming this is a hacky module alias and ignoring it. Got 'distro', module object: <module 'distro' from '/usr/lib/python3/dist-packages/distro/__init__.py'>
[mux  685121] 14:38:50.355142 D mitogen: ParentEnumerationMethod(): 'ansible.module_utils.distro' is PKG_DIRECTORY: '/usr/lib/python3/dist-packages/ansible/module_utils/distro/__init__.py'
...
[mux  685121] 14:38:51.209954 D mitogen: SysModulesMethod(): sys.modules['ansible.module_utils.distro._distro'] absent or not a regular module
[mux  685121] 14:38:51.210217 D mitogen: ParentEnumerationMethod(): imp.find_module('_distro', [['/usr/lib/python3/dist-packages/distro']]) -> No module named '_distro'
[mux  685121] 14:38:51.210346 D mitogen: get_module_source('ansible.module_utils.distro._distro'): cannot find source
[mux  685121] 14:38:51.210449 D mitogen.responder: could not find source for 'ansible.module_utils.distro._distro'
[mux  685121] 14:38:51.210566 D mitogen.responder: sending ansible.module_utils.distro._distro (0.05 KiB) to ssh.proxy.glacom.eu
[mux  685121] 14:38:51.224181 D mitogen.importer.[ssh.proxy.glacom.eu]: received ansible.module_utils.distro._distro
...
[mux  685121] 14:38:51.232206 D mitogen: MitogenProtocol(unix_client.685155): disconnecting
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/usr/lib/python3/dist-packages/ansible/executor/task_executor.py", line 663, in _execute
    result = self._handler.run(task_vars=variables)
  File "/usr/lib/python3/dist-packages/ansible_mitogen/mixins.py", line 144, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/lib/python3/dist-packages/ansible/plugins/action/normal.py", line 47, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/usr/lib/python3/dist-packages/ansible_mitogen/mixins.py", line 374, in _execute_module
    self._set_temp_file_args(module_args, wrap_async)
  File "/usr/lib/python3/dist-packages/ansible_mitogen/mixins.py", line 353, in _set_temp_file_args
    self._connection.get_good_temp_dir()
  File "/usr/lib/python3/dist-packages/ansible_mitogen/connection.py", line 817, in get_good_temp_dir
    self._connect()
  File "/usr/lib/python3/dist-packages/ansible_mitogen/connection.py", line 839, in _connect
    self._connect_stack(stack)
  File "/usr/lib/python3/dist-packages/ansible_mitogen/connection.py", line 786, in _connect_stack
    dct = mitogen.service.call(
  File "/usr/lib/python3/dist-packages/mitogen/service.py", line 126, in call
    return call_context.call_service(service_name, method_name, **kwargs)
  File "/usr/lib/python3/dist-packages/mitogen/core.py", line 2284, in call_service
    return recv.get().unpickle()
  File "/usr/lib/python3/dist-packages/mitogen/core.py", line 963, in unpickle
    raise obj
mitogen.core.CallError: builtins.ModuleNotFoundError: The Mitogen master process was unable to serve 'ansible.module_utils.distro._distro'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.
  File "<stdin>", line 3669, in _dispatch_one
  File "<stdin>", line 3656, in _parse_request
  File "<stdin>", line 668, in import_module
  File "<stdin>", line 1516, in load_module
  File "master:/usr/lib/python3/dist-packages/ansible_mitogen/target.py", line 85, in <module>
    import ansible_mitogen.runner
  File "<stdin>", line 1516, in load_module
  File "master:/usr/lib/python3/dist-packages/ansible_mitogen/runner.py", line 85, in <module>
    import ansible.module_utils.basic
  File "<stdin>", line 1516, in load_module
  File "master:/usr/lib/python3/dist-packages/ansible/module_utils/basic.py", line 152, in <module>
    from ansible.module_utils.common.sys_info import (
  File "<stdin>", line 1516, in load_module
  File "master:/usr/lib/python3/dist-packages/ansible/module_utils/common/sys_info.py", line 10, in <module>
    from ansible.module_utils import distro
  File "<stdin>", line 1516, in load_module
  File "master:/usr/lib/python3/dist-packages/ansible/module_utils/distro/__init__.py", line 45, in <module>
    from ansible.module_utils.distro import _distro as distro
  File "<stdin>", line 1491, in load_module

[aine-etke]

arch linux, ansible v2.12.2, python v3.10.2 - same issue

[stefanor]

With the Debian packaged ansible 2.10.8, mitogen 0.3.1, on Python 3.9.8. Upgrading from distro 1.6.0 to 1.7.0 on the target broke mitogen

While that's how the bug was described to me, from my own experimentation, I'd describe it as: Upgrading distro from 1.6.0 to 1.7.0 on the controller broke the use-case when the target doesn't have distro installed.

[willmerae]

Based on @stefanor's branch I think I've prototyped a more general fix willmerae@754a94b. Would anyone be willing to try it? I'll create a PR once I've teased out some refactoring from the messy investigation branch https://github.com/willmerae/mitogen/tree/issue-906.

[stefanor]

Oh, yeah, that definitely looks cleaner.