ansible fails to load submodule from upstream role linux-system-roles.network when mitogen is enabled

[aneagoe]

• Which version of Ansible are you running?

  • 2.12.2
  • 2.9.27

• Is your version of Ansible patched in any way?

  • no

• Are you running with any custom modules, or module_utils loaded?

  • yes, coming from upstream roles/collections

• Have you tried the latest master version from Git?

  • no

• Do you have some idea of what the underlying problem may be?
  https://mitogen.networkgenomics.com/ansible_detailed.html#common-problems has
  instructions to help figure out the likely cause and how to gather relevant
  logs.

  • no

• Mention your host and target OS and versions

  • control-host: Fedora 35, target-host: CentOS8 Stream, ansible 2.12.2, python 3.10.2, mitogen 0.3.2
  • control-host: CentOS8 Stream, target-host: CentOS8 Stream, ansible 2.9.27, python 3.6.8, mitogen 0.2.10

• If reporting a crash or hang in Ansible, please rerun with -vvv and include
  200 lines of output around the point of the error, along with a full copy of
  any traceback or error text in the log. Beware "-vvv" may include secret
  data! Edit as necessary before posting.

  • ansible_mitogen-a2.9-p3.6.log
  • ansible_mitogen-a2.12-p3.10.log
  • error snippet:
    [WARNING]: exception: Traceback (most recent call last): File "master:/home/andrei/.ansible/roles/linux-system-roles.network/library/network_connections.py", line 2620, in main File "master:/home/andrei/.ansible/roles/linux-system-roles.network/library/network_connections.py", line 1792, in create File "master:/home/andrei/.ansible/roles/linux-system-roles.network/library/network_connections.py", line 2008, in __init__ ModuleNotFoundError: No module named 'ansible.module_utils.network_lsr.nm'
    The same works as expected when mitogen is disabled.

• If reporting any kind of problem with Ansible, please include the Ansible
  version along with output of "ansible-config dump --only-changed".

ANSIBLE_NOCOWS(/home/andrei/code/ansible/env/prod/ansible.cfg) = True
CACHE_PLUGIN(/home/andrei/code/ansible/env/prod/ansible.cfg) = redis
CACHE_PLUGIN_CONNECTION(/home/andrei/code/ansible/env/prod/ansible.cfg) = distill-redis.example.com:30379:0:redacted
CACHE_PLUGIN_TIMEOUT(/home/andrei/code/ansible/env/prod/ansible.cfg) = 0
CALLBACKS_ENABLED(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['profile_tasks']
COLLECTIONS_PATHS(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/.ansible/collections', '/home/andrei/code/ansible/requirements_collections']
DEFAULT_ACTION_PLUGIN_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/code/ansible/env/prod/plugins/action', '/home/andrei/code/ansible/roles/ceph/plugins/actions']
DEFAULT_CALLBACK_PLUGIN_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/code/ansible/env/prod/plugins/callback', '/usr/share/ansible/plugins/callback', '/home/andrei/code/ansible/roles/ceph/plugins/callback']
DEFAULT_FILTER_PLUGIN_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/code/ansible/env/prod/plugins/filter', '/usr/share/ansible/plugins/filter', '/home/andrei/code/ansible/roles/ceph/plugins/filter']
DEFAULT_FORKS(/home/andrei/code/ansible/env/prod/ansible.cfg) = 500
DEFAULT_GATHERING(/home/andrei/code/ansible/env/prod/ansible.cfg) = implicit
DEFAULT_GATHER_TIMEOUT(/home/andrei/code/ansible/env/prod/ansible.cfg) = 60
DEFAULT_HASH_BEHAVIOUR(/home/andrei/code/ansible/env/prod/ansible.cfg) = merge
DEFAULT_HOST_LIST(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/code/ansible/env/prod/inventory']
DEFAULT_JINJA2_EXTENSIONS(/home/andrei/code/ansible/env/prod/ansible.cfg) = tt.j2ext.RaiseExtension, tt.j2ext.GitRevisionExtension
DEFAULT_MANAGED_STR(/home/andrei/code/ansible/env/prod/ansible.cfg) = This file is managed by ansible. Your changes will be overwritten.
DEFAULT_MODULE_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/code/ansible/env/prod/plugins/modules', '/home/andrei/code/ansible/roles/ceph/library']
DEFAULT_PRIVATE_KEY_FILE(/home/andrei/code/ansible/env/prod/ansible.cfg) = /home/andrei/.ssh/ansible
DEFAULT_REMOTE_USER(/home/andrei/code/ansible/env/prod/ansible.cfg) = ansible
DEFAULT_ROLES_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/code/ansible/roles', '/home/andrei/.ansible/roles', '/home/andrei/code/ansible/roles/ceph/roles', '/home/andrei/code/ansible/requirements_roles']
DEFAULT_STRATEGY(/home/andrei/code/ansible/env/prod/ansible.cfg) = mitogen_linear
DEFAULT_STRATEGY_PLUGIN_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['/home/andrei/.ansible/plugins/strategy', '/usr/share/ansible/plugins/strategy', '/usr/local/lib/python3.10/site-packages/ansible_mitogen/plugins/strategy', '/usr/lib/python3.10/site-packages/ansible_mitogen/plugins/strategy']
DEFAULT_TIMEOUT(/home/andrei/code/ansible/env/prod/ansible.cfg) = 60
HOST_KEY_CHECKING(/home/andrei/code/ansible/env/prod/ansible.cfg) = False
INTERPRETER_PYTHON(/home/andrei/code/ansible/env/prod/ansible.cfg) = /usr/bin/python3
INVALID_TASK_ATTRIBUTE_FAILED(/home/andrei/code/ansible/env/prod/ansible.cfg) = False
INVENTORY_ENABLED(/home/andrei/code/ansible/env/prod/ansible.cfg) = ['script']
RETRY_FILES_ENABLED(/home/andrei/code/ansible/env/prod/ansible.cfg) = False
RETRY_FILES_SAVE_PATH(/home/andrei/code/ansible/env/prod/ansible.cfg) = /home/andrei/code/ansible/env/prod/"~/" # The directory they will go into

[moreati]

This may have been fixed in master, by #913

[cjeanner]

Hello @moreati !
Unfortunately, I'm also hitting this issue, with a fresh "master" checkout (so I have #913).

The issue is for another module, but it seems to be really, really the same issue:

2022-03-31 04:41:33.480636 | 24420198-6297-967d-b366-000000000017 | TASK | Reserve instances
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'ansible.module_utils.openstack'
2022-03-31 04:41:33.638778 | 24420198-6297-967d-b366-000000000017 | FATAL | Reserve instances | localhost | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"master:/home/stack/mitogen/ansible_mit
ogen/runner.py\", line 975, in _run\n self._run_code(code, mod)\n File \"master:/home/stack/mitogen/ansible_mitogen/runner.py\", line 939, in _run_code\n exec(code, vars(mod))\n File \"master:/usr/share/ansible/plugins/modules/metalsmith_instances.py\", line 21, in <module>\nModuleNotFoundError: No module named 'ansible.module_utils.openstack'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Involved code is, among things:

• https://opendev.org/openstack/metalsmith/src/branch/master/metalsmith_ansible/ansible_plugins/modules/metalsmith_instances.py#L21-L28
• https://opendev.org/openstack/ansible-collections-openstack

Note that it's working just fine with the "linear" strategy from Ansible.

[amarao]

We are having the same problem with openstack modules.

Solution was to set strategy to linear explicitly for play with those modules.

[cjeanner]

It's as if mitogen doesn't know about /usr/share/ansible/collections/ansible_collections actually.
For instance, the "metalsmith_instances" module is in /usr/share/ansible/plugins/modules, and is found; but the openstack collection is, well, a collection, and its python content ends in /usr/share/ansible/collections/ansible_collections/openstack/cloud/plugins, in subdirectories.

May it be "just" that?

[moreati]

Notes to self

• https://galaxy.ansible.com/linux-system-roles/network is a single role, not an Ansible collection. It contains
  • A module https://github.com/linux-system-roles/network/blob/main/library/network_connections.py
  • Some module_utils https://github.com/linux-system-roles/network/tree/main/module_utils/
  • linux-system-roles is not a valid Python identifier
• https://galaxy.ansible.com/openstack/cloud is a collection. It contains
  • Approx 150 modules
  • 4 module_utils plugins
  • An inventory plugin (unlisted on Ansible Galaxy) https://opendev.org/openstack/ansible-collections-openstack/src/branch/master/plugins/inventory

[maxpain]

Any updates?