ModuleNotFoundError: no module named ansible_collections

[philfry]

Some ansible collections won't run with mitogen due to mitogen being unable to find the corresponding module_utils:

• community.general 2.5.1:
  • one_host
  • one_template
• netbox.netbox 3.0.0:
  • nearly all modules from this collection

How to reproduce:

(note: to test whether or not ansible/mitogen is able to find the modules it is not necessary to provide valid module attributes)

git clone https://github.com/dw/mitogen.git
cat > ansible.cfg <<EOF
[defaults]
strategy_plugins = mitogen/ansible_mitogen/plugins/strategy
strategy = mitogen_linear
EOF
ansible-galaxy install netbox.netbox community.general
cat > test-netbox.yml <<EOF
---
- hosts: localhost
  gather_facts: no
  tasks:
    - netbox.netbox.netbox_virtual_machine:
EOF
cat > test-opennebula.yml <<EOF
---
- hosts: localhost
  gather_facts: no
  tasks:
    - community.general.one_host:
EOF
ansible-playbook test-netbox.yml
ansible-playbook test-opennebula.yml

Which version of Ansible are you running?

2.10.8

Is your version of Ansible patched in any way?

no.

Are you running with any custom modules, or module_utils loaded?

no.

Have you tried the latest master version from Git?

yes.

Do you have some idea of what the underlying problem may be?

the failed modules do not include from ansible.module_utils.basic import AnsibleModule:

grep -L 'import.*AnsibleModule' ~/.ansible/collections/ansible_collections/community/general/plugins/modules/cloud/opennebula/*py
grep -L 'import.*AnsibleModule' ~/.ansible/collections/ansible_collections/netbox/netbox/plugins/modules/*py

Other modules from community.general, for example community.general.one_vm work fine as they include this line.
By adding this line to the netbox.netbox modules mitogen is able to find the required module_utils.

Mention your host and target OS and versions

• controller node: Fedora 33
• managed node: CentOS 7

Mention your host and target Python versions

• controller node: 3.9.4
• managed node: 2.7.5

[philfry]

the failed modules do not include from ansible.module_utils.basic import AnsibleModule:

looks like it doesn't matter what is being imported from ansible.module_utils als long as this library is mentioned.

Tried with netbox_virtual_machine from netbox.netbox version 3.1.1.

Case 1: Without any modifications ansible-mitogen complaints about No module named 'ansible_collections'.

Case 2: importing something from ansible.module_utils

--- netbox_virtual_machine.py
+++ netbox_virtual_machine.py
@@ -204,6 +204,8 @@
   type: str
 """
 
+from ansible.module_utils.ansible_release import __version__
+
 from ansible_collections.netbox.netbox.plugins.module_utils.netbox_utils import (
     NetboxAnsibleModule,
     NETBOX_ARG_SPEC,

=> this works.

Case 3: just adding a comment:

--- netbox_virtual_machine.py
+++ netbox_virtual_machine.py
@@ -204,6 +204,8 @@
   type: str
 """
 
+# from ansible.module_utils.foo
+
 from ansible_collections.netbox.netbox.plugins.module_utils.netbox_utils import (
     NetboxAnsibleModule,
     NETBOX_ARG_SPEC,

=> this also works. Wait, what? It doesn't even matter whether this line is commented out by # or enclosed it with """.

As I'm not deeply involved with mitogen, I'd like to get it clear:
Does mitogen scan the modules for phrases like from foo.bar and does $stuff depending on its findings (scan_fromlist() is pretty suspicious)?

[philfry]

Looks like the NewStylePlanner scans for from ansible.module_utils..

Mitogen falls back to OldStylePlanner if that line was not found. And afaict the OldStylePlanner doesn't do scan_fromlist and stuff – and therefore fails with collections.

[Zocker1999NET]

@philfry I saw you implemented a fix. Does it work flawless? And, if so, do you mind creating a pull request? Nevermind, did not saw the PR #844 from you

[garthwilliamson]

I've tested the PR #844 and it fixes the issue for us so far.