New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_region_facts module requires region parameter with the latest boto3 python module #40511

Closed
ansiblejunky opened this Issue May 21, 2018 · 12 comments

Comments

Projects
None yet
6 participants
@ansiblejunky
Copy link

ansiblejunky commented May 21, 2018

SUMMARY

Using the aws_region_facts module to retrieve all regions, I get the following error message that the region parameter is required. Our Ansible documentation says it is optional. In fact it shouldn't be required because you want to get all regions. However it seems the latest boto3 module requires it.

ISSUE TYPE
  • Bug Report
COMPONENT NAME

aws_region_facts

ANSIBLE VERSION
ansible 2.5.0
  config file = /var/lib/awx/projects/test/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
CONFIGURATION
SHOW_CUSTOM_STATS(/var/lib/awx/projects/test/ansible.cfg) = True
OS / ENVIRONMENT

RHEL 7.5
Ansible 2.5.0
Python 2.7.5
boto3 version...

# python2
Python 2.7.5 (default, Feb 20 2018, 09:19:12)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> print boto3.__version__
1.4.4
STEPS TO REPRODUCE
---
- name: Cleanup all AWS instances, snapshots, etc.
  hosts: localhost
  connection: local
  gather_facts: false

   tasks:
      - name: Gather facts about all regions
        aws_region_facts:
EXPECTED RESULTS

Module returns all regions.

ACTUAL RESULTS

I receive an error that the region parameter is required. However, documentation states that it is not required. This is not a documentation problem but really it is an issue with the latest boto3 Python module it seems.

# ansible-playbook -i inventory/inventory_local aws.yml

PLAY [Cleanup all AWS instances, snapshots, etc.] ******************************************

TASK [aws_region_facts] ********************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "The aws_region_facts module requires a region and none was found in configuration, environment variables or module parameters"}
	to retry, use: --limit @/var/lib/awx/projects/test/aws.retry

PLAY RECAP *********************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

[root@localhost test]# clear
[root@localhost test]# ansible-playbook -i inventory/inventory_local aws.yml -vvvv
ansible-playbook 2.5.0
  config file = /var/lib/awx/projects/test/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.5 (default, Feb 20 2018, 09:19:12) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
Using /var/lib/awx/projects/test/ansible.cfg as config file
setting up inventory plugins
Set default localhost to localhost
Parsed /var/lib/awx/projects/test/inventory/inventory_local inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: aws.yml **************************************************************************
1 plays in aws.yml

PLAY [Cleanup all AWS instances, snapshots, etc.] ******************************************
META: ran handlers

TASK [aws_region_facts] ********************************************************************
task path: /var/lib/awx/projects/test/aws.yml:9
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/amazon/aws_region_facts.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636 `" && echo ansible-tmp-1526939120.78-146000614122636="` echo /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636 `" ) && sleep 0'
<localhost> PUT /root/.ansible/tmp/ansible-local-21853hpsShl/tmpPib3h7 TO /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636/aws_region_facts.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636/ /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636/aws_region_facts.py && sleep 0'
<localhost> EXEC /bin/sh -c 'python /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636/aws_region_facts.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1526939120.78-146000614122636/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/tmp/ansible_z_kX1m/ansible_modlib.zip/ansible/module_utils/ec2.py", line 106, in boto3_conn
    return _boto3_conn(conn_type=conn_type, resource=resource, region=region, endpoint=endpoint, **params)
  File "/tmp/ansible_z_kX1m/ansible_modlib.zip/ansible/module_utils/ec2.py", line 129, in _boto3_conn
    client = boto3.session.Session(profile_name=profile).client(resource, region_name=region, endpoint_url=endpoint, **params)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/session.py", line 836, in create_client
    client_config=config, api_version=api_version)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/client.py", line 71, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/client.py", line 281, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/args.py", line 45, in get_client_args
    endpoint_url, is_secure, scoped_config)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/args.py", line 110, in compute_client_args
    service_name, region_name, endpoint_url, is_secure)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/client.py", line 354, in resolve
    service_name, region_name)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/regions.py", line 122, in construct_endpoint
    partition, service_name, region_name)
  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()

fatal: [localhost]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_secret_key": null,
            "ec2_url": null,
            "filters": {},
            "profile": null,
            "region": null,
            "security_token": null,
            "validate_certs": true
        }
    },
    "msg": "The aws_region_facts module requires a region and none was found in configuration, environment variables or module parameters"
}
	to retry, use: --limit @/var/lib/awx/projects/test/aws.retry

PLAY RECAP *********************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

The problem appears to be with botocore/regions.py on line 135 where it requires a region value.

  File "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/botocore/regions.py", line 135, in _endpoint_for_partition
    raise NoRegionError()

@ansiblejunky ansiblejunky changed the title `aws_region_facts` module requires a `region` with the latest `boto3` module aws_region_facts module requires region parameter with the latest boto3 python module May 21, 2018

@ansibot

This comment has been minimized.

Copy link
Contributor

ansibot commented May 21, 2018

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibot

This comment has been minimized.

Copy link
Contributor

ansibot commented May 21, 2018

@maxamillion

This comment has been minimized.

Copy link
Contributor

maxamillion commented May 22, 2018

cc @ryansb

@ryansb

This comment has been minimized.

Copy link
Contributor

ryansb commented May 23, 2018

I'm unable to reproduce on either 2.5.3 or 2.6-dev. It seems this issue has been fixed in later 2.5.x versions.

ansible --version && ansible localhost -c local -m aws_region_facts -a 'aws_access_key=BLEEP aws_secret_key="BLEEP"'                                   
ansible 2.5.3                                       
  config file = None      
  configured module search path = [u'/home/ryansb/.ansible/plugins/modules', u'/usr/share/ansible/plugins
/modules']                                                                                               
  ansible python module location = /home/ryansb/.pyenv/versions/2.7.12/envs/tmpenv-2CImOMApA2fA3Nd/lib/py
thon2.7/site-packages/ansible
  executable location = /home/ryansb/.pyenv/versions/tmpenv-2CImOMApA2fA3Nd/bin/ansible
  python version = 2.7.12 (default, Sep 27 2016, 18:08:33) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)]        
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source
                          
 [WARNING]: No inventory was parsed, only implicit localhost is available
                                                                                                         
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'      
                          
localhost | SUCCESS => {                                                                                 
    "changed": false,                               
    "regions": [          
        {                 
            "endpoint": "ec2.ap-south-1.amazonaws.com",                                                  
            "region_name": "ap-south-1"             
        },
.....
$ ansible --version && ansible localhost -c local -m aws_region_facts -a 'aws_access_key=BLEEP aws_secret_key="BLEEP"'                                   
ansible 2.6.0dev0                                   
  config file = None      
  configured module search path = [u'/home/ryansb/.ansible/plugins/modules', u'/usr/share/ansible/plugins
/modules']                                                                                               
  ansible python module location = /home/ryansb/.pyenv/versions/2.7.12/envs/tmpenv-mVWRYKD3D6kUIj3/lib/py
thon2.7/site-packages/ansible
  executable location = /home/ryansb/.pyenv/versions/tmpenv-mVWRYKD3D6kUIj3/bin/ansible
  python version = 2.7.12 (default, Sep 27 2016, 18:08:33) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)]        
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source
                          
 [WARNING]: No inventory was parsed, only implicit localhost is available
                                                                                                         
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'      
                          
localhost | SUCCESS => {                                                                                 
    "changed": false,                               
    "regions": [          
        {                 
            "endpoint": "ec2.ap-south-1.amazonaws.com",                                                  
            "region_name": "ap-south-1"             
        },
.....

@ryansb ryansb closed this May 23, 2018

@GoodMirek

This comment has been minimized.

Copy link

GoodMirek commented May 26, 2018

Happens to me to, with ansible 2.5.3.

ansible --version
ansible 2.5.3
  config file = /home/ubuntu/.ansible.cfg
  configured module search path = [u'/home/ubuntu/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

The boto3 comes from Ubuntu 18.04 repo.
What else could I provide to help troubleshooting?

@ansiblejunky

This comment has been minimized.

Copy link
Author

ansiblejunky commented May 29, 2018

@GoodMirek Thanks! Please provide the boto3 version on your system. Run pip list perhaps.

@ansiblejunky

This comment has been minimized.

Copy link
Author

ansiblejunky commented May 29, 2018

@ryansb Your test is incomplete, because we need to know the boto3 version that you have installed on your system. Ansible 2.5.3 might have worked for you because you have an older boto3 installed on your system. Can you check that please?

@ryansb

This comment has been minimized.

Copy link
Contributor

ryansb commented May 29, 2018

$ pip list | grep boto
boto3           1.7.28  
botocore        1.10.28
@ryansb

This comment has been minimized.

Copy link
Contributor

ryansb commented May 29, 2018

Also works with:

pip list | grep boto
boto3           1.5.0   
botocore        1.8.50

It also works on boto3 v1.3.0, which was released in March 2016, more than 2 years ago. This isn't an ansible bug.

@ansiblejunky

This comment has been minimized.

Copy link
Author

ansiblejunky commented May 29, 2018

Thanks, I will try to upgrade my boto3 and botocore and test again. I was only thinking this was a documentation issue - that we would mention the region becomes required but if recent versions of botoXXX fixes the issue, then it is not necessary.

@GoodMirek

This comment has been minimized.

Copy link

GoodMirek commented Jun 4, 2018

UPDATE: I use instance profile for obtaining credentials.

Does not work for me with the latest versions. Ansible node is Ubuntu 16.04, instrumented node is Ubuntu 18.04.
Instrumented node:

pip list --format=columns | grep boto
boto3           1.7.31
botocore        1.10.31

The ansible node:

ansible --version
ansible 2.5.4
  config file = /home/ubuntu/.ansible.cfg
  configured module search path = [u'/home/ubuntu/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

Error printed by ansible while running playbook:

TASK [Gather current region EC2 endpoint] *******************************************************************************************************************
fatal: [172.31.17.123]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 172.31.17.123 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_Ds9aqf/ansible_module_aws_region_facts.py\", line 112, in <module>\r\n    main()\r\n  File \"/tmp/ansible_Ds9aqf/ansible_module_aws_region_facts.py\", line 94, in main\r\n    sanitized_filters = dict((k.replace('_', '-'), v) for k, v in module.params.get('filters').items())\r\nAttributeError: 'NoneType' object has no attribute 'items'\r\n", "msg": "MODULE FAILURE", "rc": 1}
fatal: [172.31.17.122]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 172.31.17.122 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_9oiCiw/ansible_module_aws_region_facts.py\", line 112, in <module>\r\n    main()\r\n  File \"/tmp/ansible_9oiCiw/ansible_module_aws_region_facts.py\", line 94, in main\r\n    sanitized_filters = dict((k.replace('_', '-'), v) for k, v in module.params.get('filters').items())\r\nAttributeError: 'NoneType' object has no attribute 'items'\r\n", "msg": "MODULE FAILURE", "rc": 1}

Error printed by ansible while invoking the module from command line:

ansible all -m aws_region_facts
172.31.17.123 | FAILED! => {
    "changed": false,
    "msg": "The aws_region_facts module requires a region and none was found in configuration, environment variables or module parameters"
}
172.31.17.122 | FAILED! => {
    "changed": false,
    "msg": "The aws_region_facts module requires a region and none was found in configuration, environment variables or module parameters"
}

With region parameter it works correctly in both playbook and while invoked from command line:

ansible all -m aws_region_facts -a 'region=eu-central-1'
172.31.17.123 | SUCCESS => {
    "changed": false, 
    "regions": [
        {
            "endpoint": "ec2.ap-south-1.amazonaws.com", 
            "region_name": "ap-south-1"
        }, 
  <OUTPUT TRUNCATED>
@ansiblejunky

This comment has been minimized.

Copy link
Author

ansiblejunky commented Aug 3, 2018

@ryansb I suggest we re-open this issue as a result of the info from @GoodMirek. The boto3 and botocore versions he had were more recent that yours, and as a result hit the same issue I did. The problem is more with the fact that boto3 and botocore now require the region parameter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment