"Boto3 required for this module" when using delegate_to

[knightsg]

Got another fun one for you :)

First up, info:

Linux 3.13.0-164-generic #214-Ubuntu SMP Wed Dec 5 10:42:33 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

ansible-playbook 2.6.12.post0 (stable-2.6 114d5c0368) last updated 2019/01/31 22:30:45 (GMT +000)
  config file = /var/jenkins/.ansible.cfg
  configured module search path = [u'/var/jenkins/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/ansible/ansible-2.6/lib/ansible
  executable location = /opt/ansible/ansible-2.6/bin/ansible-playbook
  python version = 2.7.6 (default, Nov 13 2018, 12:45:42) [GCC 4.8.4]

I believe I noticed this while I was helping troubleshoot the issue in #499, so it predates the fixes that were committed for that ticket. However, it's still happening for me since I pulled the latest master commits. We run ansible from a bunch of jenkins agents in various AWS VPCs, and when I run ansible on these hosts with mitogen enabled it fails at the following task, with the error message "Boto3 required for this module":

- name: gather ec2 facts for nagios
  become: False
  ec2_instance_facts:
    region: "{{ nagios_rds_region|default('us-east-1') }}"
    filters:
      instance-state-name: running
  delegate_to: 127.0.0.1
  register: nagios_ec2_instances

When I disable mitogen the task runs without error.

[dw]

Iinteresting :) How is boto installed? I think the magic here is "127.0.0.1" -- it is different (believe it or not) from the magic Ansible "localhost". The default transport is used -- so this is running as an SSH connection to the local machine, rather than as a simple subprocess using the inherited Python/virtualenv/whatever environment of the controller.

But in either case, it's a difference and we need to fix it.

Do you know how and where boto is installed on that machine? e.g. Debian package, virtualenv, etc. Also do you know which interpreter it is installed for? Master has subtle changes to python_path which I need to review before 0.2.4 is out -- it could be that Ansible is running with a different interpreter than the one used for '127.0.0.1'

[knightsg]

It's installed with pip:

$ pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)

$ pip freeze
boto==2.49.0
boto3==1.9.64
botocore==1.12.64

I'll try changing to localhost and see if that works.

[knightsg]

I changed 127.0.0.1 to localhost and ran the playbook but it still fails with the same error unfortunately.

[dw]

This might related to https://github.com/dw/mitogen/issues/476 -- does setting "-e ansible_python_interpreter=python" have any effect? Also try with "/usr/bin/python".

[dw]

0.2.5 will include this before the end of the week.

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!

[akshayhiremath]

@dw https://github.com/dw/mitogen/issues/476 is not accessible. GIves error "This issue has been moved to a repository you don't have access to."