AWS Route53 - ModuleNotFoundError: No module named 'ansible_collections' Error

[ccravens]

Which version of Ansible are you running?
ansible 2.10.3
config file = ansible.cfg
configured module search path = ['/Users/ccravens/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/Cellar/ansible/2.10.4/libexec/lib/python3.9/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.9.1 (default, Dec 10 2020, 10:36:41) [Clang 11.0.0 (clang-1100.0.33.17)]

Is your version of Ansible patched in any way?
No... Not that I know of. I did install the route53 with galaxy: ansible-galaxy collection install community.aws
https://docs.ansible.com/ansible/latest/collections/community/aws/route53_module.html

Are you running with any custom modules, or module_utils loaded?
No

Have you tried the latest master version from Git?
Yes, both master and v0.3.0-rc.0

Do you have some idea of what the underlying problem may be?
None, but I think it may be due to Ansible 2.10

Mention your host and target OS and versions
Mac OSX - running local

Mention your host and target Python versions
Same as above, running local

If reporting a crash or hang in Ansible, please rerun with -vvv and include

TASK [ingress : Set DNS Record to Load Balancer FQDN] ********************************************************************************************
task path: /Users/ccravens/Business/ossys/Projects/Platform/src/platform/k8s/ansible/deploy/roles/ingress/tasks/main.yaml:25
[task 9827] 01:39:50.781193 D ansible_mitogen.process: will use multiplexer 0 (/var/folders/wk/gmnv4rxn2v7crq1npr593bz40000gn/T/mitogen_unix_x2tyjniy.sock) to connect to "127.0.0.1"
[task 9827] 01:39:50.782204 D mitogen.unix: client: connecting to /var/folders/wk/gmnv4rxn2v7crq1npr593bz40000gn/T/mitogen_unix_x2tyjniy.sock
[task 9827] 01:39:50.782989 D mitogen.unix: client: local ID is 1005, remote is 0
[mux  9784] 01:39:50.782998 D mitogen.unix: listener: accepted connection from PID 9827: unix_client.9827
[task 9827] 01:39:50.790155 D ansible_mitogen.planner: <class 'ansible_mitogen.planner.BinaryPlanner'> rejected 'community.aws.route53'
[task 9827] 01:39:50.790708 D ansible_mitogen.planner: <class 'ansible_mitogen.planner.NewStylePlanner'> rejected 'community.aws.route53'
[task 9827] 01:39:50.791134 D ansible_mitogen.planner: <class 'ansible_mitogen.planner.JsonArgsPlanner'> rejected 'community.aws.route53'
[task 9827] 01:39:50.791538 D ansible_mitogen.planner: <class 'ansible_mitogen.planner.WantJsonPlanner'> rejected 'community.aws.route53'
[task 9827] 01:39:50.791926 D ansible_mitogen.planner: <class 'ansible_mitogen.planner.OldStylePlanner'> accepted 'community.aws.route53' (filename '/Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py')
[mux  9784] 01:39:50.792915 D mitogen.service: caching small file /Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py
[mux  9784] 01:39:50.793442 D mitogen.service: requesting Context(2, 'local.9786') cache and forward small file to Context(2, 'local.9786'): /Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py
[mux  9784] 01:39:50.794607 D mitogen.service.[local.9786]: PushFileService().store_and_forward('/Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py', [blob: 24248 bytes], Context(2, 'local.9786')) 'mitogen.Pool.44a8.1'
[task 9827] 01:39:50.794744 D mitogen.parent: starting function call to local.9786: ansible_mitogen.target.run_module(kwargs={'runner_name': 'OldStyleRunner', 'module': 'community.aws.route53', 'path': '/Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py', 'json_args': '{"profile": "dns", "state": "present", "zone": "analyticshq.com", "record": "*.dev.analyticshq.com", "alias_hosted_zone_id": "Z166TLBEWOO7G0", "type": "A", "alias": true, "ttl": 300, "overwrite": true, "value": "a8c094325f2cf4339ad2522d61d810c1-1746710274.us-gov-east-1.elb.amazonaws.com", "wait": true, "_ansible_check_mode": false, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 3, "_ansible_version": "2.10.3", "_ansible_module_name": "community.aws.route53", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}', 'env': {'KUBECONFIG': '/Users/ccravens/.kube/platform/aws.us-gov-east-1.dev.config'}, 'interpreter_fragment': 'python3', 'is_python': False, 'good_temp_dir': '/Users/ccravens/.ansible/tmp', 'cwd': '/Users/ccravens/Business/ossys/Projects/Platform/src/platform/k8s/ansible/deploy', 'extra_env': {}, 'emulate_tty': True, 'service_context': Context(0, None)})
[mux  9784] 01:39:50.796247 D mitogen.[local.9786]: Dispatcher: dispatching ('Chads-MBP-2028.localdomain-9827-108ea35c0-5eea3e', 'ansible_mitogen.target', None, 'run_module', (), Kwargs({'kwargs': {'runner_name': 'OldStyleRunner', 'module': 'community.aws.route53', 'path': '/Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py', 'json_args': '{"profile": "dns", "state": "present", "zone": "analyticshq.com", "record": "*.dev.analyticshq.com", "alias_hosted_zone_id": "Z166TLBEWOO7G0", "type": "A", "alias": true, "ttl": 300, "overwrite": true, "value": "a8c094325f2cf4339ad2522d61d810c1-1746710274.us-gov-east-1.elb.amazonaws.com", "wait": true, "_ansible_check_mode": false, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 3, "_ansible_version": "2.10.3", "_ansible_module_name": "community.aws.route53", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}', 'env': {'KUBECONFIG': '/Users/ccravens/.kube/platform/aws.us-gov-east-1.dev.config'}, 'interpreter_fragment': 'python3', 'is_python': False, 'good_temp_dir': '/Users/ccravens/.ansible/tmp', 'cwd': '/Users/ccravens/Business/ossys/Projects/Platform/src/platform/k8s/ansible/deploy', 'extra_env': {}, 'emulate_tty': True, 'service_context': Context(0, 'master')}}))
[mux  9784] 01:39:50.797602 D ansible_mitogen.target.[local.9786]: exec_args(['/bin/sh', '-c', 'python3 /Users/ccravens/.ansible/tmp/ansible_mitogen_runner_slx5gbwe/route53.py /Users/ccravens/.ansible/tmp/ansible_mitogen_runner_slx5gbwe/ansible_mitogenpczsme3l-args'], ..., chdir=None)
[mux  9784] 01:39:51.026698 D mitogen.[local.9786]: Dispatcher: Message(2, 1005, 0, 101, 1002, b'\x80\x02(X0\x00\x00\x00Chads-MBP-2028.localdomain-9827-108ea35c0-'..1635) -> {'rc': 1, 'stdout': 'Traceback (most recent call last):\r\n  File "/Users/ccravens/.ansible/tmp/ansible_mitogen_runner_slx5gbwe/route53.py", line 378, in <module>\r\n    from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule\r\nModuleNotFoundError: No module named \'ansible_collections\'\r\n', 'stderr': ''}
[task 9827] 01:39:51.027450 D ansible_mitogen.connection: Call took 232 ms: ansible_mitogen.target.run_module(kwargs={'runner_name': 'OldStyleRunner', 'module': 'community.aws.route53', 'path': '/Users/ccravens/.ansible/collections/ansible_collections/community/aws/plugins/modules/route53.py', 'json_args': '{"profile": "dns", "state": "present", "zone": "analyticshq.com", "record": "*.dev.analyticshq.com", "alias_hosted_zone_id": "Z166TLBEWOO7G0", "type": "A", "alias": true, "ttl": 300, "overwrite": true, "value": "a8c094325f2cf4339ad2522d61d810c1-1746710274.us-gov-east-1.elb.amazonaws.com", "wait": true, "_ansible_check_mode": false, "_ansible_no_log": false, "_ansible_debug": false, "_ansible_diff": false, "_ansible_verbosity": 3, "_ansible_version": "2.10.3", "_ansible_module_name": "community.aws.route53", "_ansible_syslog_facility": "LOG_USER", "_ansible_selinux_special_fs": ["fuse", "nfs", "vboxsf", "ramfs", "9p", "vfat"], "_ansible_string_conversion_action": "warn", "_ansible_socket": null, "_ansible_shell_executable": "/bin/sh", "_ansible_keep_remote_files": false, "_ansible_tmpdir": null, "_ansible_remote_tmp": "~/.ansible/tmp"}', 'env': {'KUBECONFIG': '/Users/ccravens/.kube/platform/aws.us-gov-east-1.dev.config'}, 'interpreter_fragment': 'python3', 'is_python': False, 'good_temp_dir': '/Users/ccravens/.ansible/tmp', 'cwd': '/Users/ccravens/Business/ossys/Projects/Platform/src/platform/k8s/ansible/deploy', 'extra_env': {}, 'emulate_tty': True, 'service_context': Context(0, None)})
[task 9827] 01:39:51.028790 D ansible_mitogen.mixins: _remove_tmp_path(None)
[task 9827] 01:39:51.029221 D ansible_mitogen.mixins: _remove_tmp_path(None)
[task 9827] 01:39:51.029685 D mitogen.parent: starting no-reply function call to 'local.9786': mitogen.core.Dispatcher.forget_chain('Chads-MBP-2028.localdomain-9827-108ea35c0-5eea3e')
[mux  9784] 01:39:51.030513 D ansible_mitogen.services: decrementing reference count for Context(2, 'local.9786')
[mux  9784] 01:39:51.031069 D mitogen.[local.9786]: Dispatcher: dispatching (None, 'mitogen.core', 'Dispatcher', 'forget_chain', ('Chads-MBP-2028.localdomain-9827-108ea35c0-5eea3e',), Kwargs({}))
[mux  9784] 01:39:51.031470 D mitogen.[local.9786]: Dispatcher: Message(2, 1005, 0, 101, 0, b'\x80\x02(NX\x0c\x00\x00\x00mitogen.coreq\x00X\n\x00\x00\x00Dispatcherq\x01X\x0c\x00\x00\x00forge'..153) -> None
[task 9827] 01:39:51.032117 D mitogen: MitogenProtocol(unix_listener.9784): disconnecting
[mux  9784] 01:39:51.032722 D mitogen: <Side of unix_client.9827 fd 107>: empty read, disconnecting
[task 9827] 01:39:51.032986 D mitogen: Waker(fd=48/49): disconnecting
[mux  9784] 01:39:51.033175 D mitogen: MitogenProtocol(unix_client.9827): disconnecting
[task 9827] 01:39:51.033626 D mitogen: Router(Broker(6be0)): stats: 0 module requests in 0 ms, 0 sent (0 ms minify time), 0 negative responses. Sent 0.0 kb total, 0.0 kb avg.
The full traceback is:
Traceback (most recent call last):
  File "/Users/ccravens/.ansible/tmp/ansible_mitogen_runner_slx5gbwe/route53.py", line 378, in <module>
    from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule
ModuleNotFoundError: No module named 'ansible_collections'
fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "module_stderr": "",
    "module_stdout": "Traceback (most recent call last):\r\n  File \"/Users/ccravens/.ansible/tmp/ansible_mitogen_runner_slx5gbwe/route53.py\", line 378, in <module>\r\n    from ansible_collections.amazon.aws.plugins.module_utils.core import AnsibleAWSModule\r\nModuleNotFoundError: No module named 'ansible_collections'\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ***************************************************************************************************************************************
127.0.0.1                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

[mux  9784] 01:39:51.042316 D mitogen: Waker(fd=38/40): disconnecting
[mux  9784] 01:39:51.043068 D mitogen.service: thread mitogen.Pool.db50.21 exiting gracefully
[mux  9784] 01:39:51.043577 D mitogen.service: thread mitogen.Pool.db50.18 exiting gracefully

If reporting any kind of problem with Ansible, please include the Ansible version along with output of "ansible-config dump --only-changed"

(base) Chads-MBP-2028:k8s ccravens$ ansible-config dump --only-changed
DEFAULT_STRATEGY(/Users/ccravens/Business/ossys/Projects/Platform/src/platform/k8s/ansible/ansible.cfg) = mitogen_linear
DEFAULT_STRATEGY_PLUGIN_PATH(/Users/ccravens/Business/ossys/Projects/Platform/src/platform/k8s/ansible/ansible.cfg) = ['/Users/ccravens/Business/o
HOST_KEY_CHECKING(env: ANSIBLE_HOST_KEY_CHECKING) = False

[badfiles]

should this also happen with any collection?

[mux  39653] 15:04:30.305273 D mitogen.[****.sudo.postgres]: Dispatcher: Message(5, 2091, 0, 101, 1003, b'\x80\x02(X"\x00\x00\x00vlad-40202-7f348108f740-24ce69cb0aq\x00X\x16\x00\x00\x00a'..2339) -> {'rc': 1, 'stdout': '', 'stderr': 'Traceback (most recent call last):
 File "master:/usr/local/lib/python3.8/dist-packages/ansible_mitogen/runner.py", line 975, in _run
    self._run_code(code, mod)
  File "master:/usr/local/lib/python3.8/dist-packages/ansible_mitogen/runner.py", line 939, in _run_code
    exec(code, vars(mod))
 File "master:/home/***/dev/ansible/collections/ansible_collections/community/general/plugins/modules/postgresql_user.py", line 273, in <module>
ModuleNotFoundError: No module named \'ansible_collections.community.general.plugins\'
'}

please confirm that any collection support is still broken in 522085a

[s1113950]

@badfiles do you have that collection installed? Collection support was added in #715. Does it work for you without Mitogen?

[badfiles]

yes, w/o mitogen collections work great
we have collections in common code repo, so collections appear locally in ../collections/ansible_collections/ while playbooks are in ../

[codefriar]

I too, am experiencing this. Using ansible 2.10.6, Mitogen 0.3.0rc1. Attempting to use community.general.docker_compose: in a role. Results in:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'ansible_collections'
fatal: [myserver]: FAILED! => {"changed": false, "module_stderr": "", "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/kpoorman/.ansible/tmp/ansible_mitogen_runner_t9dtaaxl/docker_compose.py\", line 485, in <module>\r\n    from ansible_collections.community.docker.plugins.module_utils.common import (\r\nModuleNotFoundError: No module named 'ansible_collections'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

[codefriar]

yes, w/o mitogen collections work great
we have collections in common code repo, so collections appear locally in ../collections/ansible_collections/ while playbooks are in ../

out of curiosity, are you running ansible in a python virtual env? I am, and I'm wondering if mitogen is looking in the wrong spot for the collection?

[ccravens]

hmmm good point, yes I believe I'm running a python 3.8 that's installed by anaconda

[codefriar]

I’m running a 3.7, but the venv is in /opt/venv/ I wonder where mitogen is looking for collections?

…

On Feb 5, 2021, 2:45 PM -0500, Chad Cravens ***@***.***>, wrote: hmmm good point, yes I believe I'm running a python 3.8 that's installed by anaconda — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[badfiles]

i'm running 3.8 installed as regular system binary

[s1113950]

@badfiles @codefriar It's possible Mitogen is looking in the wrong place then. It uses list_collection_dirs from Ansible here: https://github.com/ansible/ansible/blob/v2.10.0/lib/ansible/collections/list.py#L49 to find collections to use during job runs.

[badfiles]

yep, setting https://docs.ansible.com/ansible/latest/reference_appendices/config.html#collections-paths
in config helped, thanks

[darkspadez]

@s1113950 Is it possible to auto search the folder collections? Currently, Ansible allows this and works with it as per https://docs.ansible.com/ansible/latest/user_guide/collections_using.html#installing-collections

Without mitogen it works w/o error, with mitogen it complains that it cannot find the collection.

Adding: collections_paths = ~/.ansible/collections:/usr/share/ansible/collections:collections fixes the issue but if the default paths change I will also need to adjust that. If you need more information please let me know.

[s1113950]

It should be possible I think. Mitogen finds collections via the Ansible function list_collection_dirs() here: https://github.com/mitogen-hq/mitogen/blob/v0.3.0rc1/ansible_mitogen/planner.py#L569 , maybe there's a config setting we need to set to have it autosearch the collections folder 🤔

[gertvdijk]

I'm also affected by the inability to find any collection (including those bundled) when installed in a venv. I noticed the collections-scan-sys-path (default True) which allows for this, I presume, but fails with Mitogen installed (0.3.0rc1).

Having it to point to an installation-specific path (different for every venv) feels very wrong. Also, it doesn't work for me when using e.g. community.docker.docker_network applied on a remote machine.

ANSIBLE_COLLECTIONS_PATHS=~/.local/share/virtualenvs/ansible-onxPID3p/lib/python3.9/site-packages/ansible_collections ansible-playbook ...

still gives me the ModuleNotFoundError: No module named 'ansible_collections'

The full traceback is:                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                                       
  File "/root/.ansible/tmp/ansible_mitogen_runner_fxo0eidb/docker_network.py", line 285, in <module>                                                                                                                                     
    from ansible_collections.community.general.plugins.module_utils.docker.common import (                                                                                                                                               
ModuleNotFoundError: No module named 'ansible_collections'                                                                                                                                                                               
fatal: [myhost]: FAILED! => changed=false                                                                                                                                                                                                 
  module_stderr: ''                                                                                                                                                                                                                      
  module_stdout: |-                                                                                                                                                                                                                      
    Traceback (most recent call last):                                                                                                                                                                                                   
      File "/root/.ansible/tmp/ansible_mitogen_runner_fxo0eidb/docker_network.py", line 285, in <module>
        from ansible_collections.community.general.plugins.module_utils.docker.common import (                                                                                                                                           
    ModuleNotFoundError: No module named 'ansible_collections'                                                  
  msg: |-                                                                                                                                                                                                                                
    MODULE FAILURE                                                                                                                                                                                                                       
    See stdout/stderr for the exact error                                                                           
  rc: 1                                                                                                             

This fails as if ansible_collections should be installed on the target machine, but this should not be the case (successful without Mitogen).

ansible-galaxy collection list lists the collection just fine too:

$ ansible-galaxy collection list                                                                                                                                                                     
                                                                                                                                                                                                                                         
# /home/gert/.local/share/virtualenvs/ansible-onxPID3p/lib/python3.9/site-packages/ansible_collections              
Collection                Version                                                                                                                                                                                                        
------------------------- -------
[...]
community.docker          1.2.2
[...]
community.general         1.3.6
[...]

Update: I managed to solve this issue by upgrading the ansible package to 3.3.0 (with ansible-base being 2.10.8). 🎉

[maxpain]

Any updates?