EC2 user-data error when spot_price != null

[brainstorm]

ISSUE TYPE

• Bug Report

COMPONENT NAME

ec2

ANSIBLE VERSION

$ ansible --version
ansible 2.4.2.0
  config file = /Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg
  configured module search path = ['/Users/romanvg/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible
  executable location = /Users/romanvg/.miniconda/envs/ansible24/bin/ansible
  python version = 3.6.3 | packaged by conda-forge | (default, Nov  4 2017, 10:13:32) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]

CONFIGURATION

$ ansible-config dump --only-changed
ANSIBLE_SSH_ARGS(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = -o ControlMaster=auto
HOST_KEY_CHECKING(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = False
INVENTORY_ENABLED(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = ['host_list', 'script', 'yaml', 'ini', 'constructed']
RETRY_FILES_ENABLED(/Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg) = False

OS / ENVIRONMENT

N/A

SUMMARY

There's a parsing error on the base64 encoding of user-data when spot-price has a defined price.

https://github.com/brainstorm/pcgr-deploy/blob/master/ansible/aws.yml#L16

STEPS TO REPRODUCE

ansible-playbook aws.yaml

---
- hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Launch EC2 instance
      ec2:
         group: "{{ aws.security_group }}"
         instance_type: "{{ aws.instance_type }}"
         instance_profile_name: "{{ aws.instance_profile_name }}"
         instance_tags:
             Name: pcgr "{{ pcgr.release }}"
             name: pcgr
             pcgrexecid: "{{ pcgr.execution_id }}"
         spot_price: "{{ aws.spot_price }}"
         user_data: "{{ lookup('file', 'files/s3_user_data.sh') }}"
         image: "{{ aws.image_id }}"
         keypair: "{{ aws.keypair }}"
         region: "{{ aws.region }}"
         zone: "{{ aws.zone }}"
         wait: yes
         volumes:
           - device_name: "{{ aws.volume_device }}"
             volume_type: gp2
             volume_size: "{{ aws.volume_size }}"
             delete_on_termination: true
         count_tag:
           name: pcgr
         exact_count: 1
      register: ec2

    - name: Add new instance to host group
      add_host:
        hostname: "{{ item.public_dns_name }}"
        groupname: launched
      with_items: "{{ ec2.instances }}"

    - name: Wait for SSH to come up
      wait_for:
        host: "{{ item.public_dns_name }}"
        port: 22
        delay: 60
        timeout: 320
        state: started
      with_items: "{{ ec2.instances }}"

EXPECTED RESULTS

Clean instantiation of an ec2 instance, loading its corresponding user-data:

https://github.com/brainstorm/pcgr-deploy/blob/master/ansible/files/s3_user_data.sh

ACTUAL RESULTS

(ansible24) brainstorm:ansible romanvg$ ansible-playbook aws.yml -vvvv
ansible-playbook 2.4.2.0
  config file = /Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg
  configured module search path = ['/Users/romanvg/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible
  executable location = /Users/romanvg/.miniconda/envs/ansible24/bin/ansible-playbook
  python version = 3.6.3 | packaged by conda-forge | (default, Nov  4 2017, 10:13:32) [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
Using /Users/romanvg/dev/pcgr-deploy/ansible/ansible.cfg as config file
setting up inventory plugins
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: Could not match supplied host pattern, ignoring: all

 [WARNING]: provided hosts list is empty, only localhost is available

Loading callback plugin default of type stdout, v2.0 from /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible/plugins/callback/__init__.py

PLAYBOOK: aws.yml ************************************************************************************************************************************************************************************************************************************************
1 plays in aws.yml

PLAY [localhost] *************************************************************************************************************************************************************************************************************************************************
META: ran handlers

TASK [Launch EC2 instance] ***************************************************************************************************************************************************************************************************************************************
task path: /Users/romanvg/dev/pcgr-deploy/ansible/aws.yml:6
Using module file /Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/ansible/modules/cloud/amazon/ec2.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: romanvg
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460 `" && echo ansible-tmp-1516170269.445788-236322726281460="` echo /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460 `" ) && sleep 0'
<127.0.0.1> PUT /var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/tmpk8nz061i TO /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ec2.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ec2.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/romanvg/.miniconda/envs/ansible24/bin/python /Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/ec2.py; rm -rf "/Users/romanvg/.ansible/tmp/ansible-tmp-1516170269.445788-236322726281460/" > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 1725, in <module>
    main()
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 1719, in main
    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 963, in enforce_count
    = create_instances(module, ec2, vpc, override_count=to_create)
  File "/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py", line 1253, in create_instances
    res = ec2.request_spot_instances(spot_price, **params)
  File "/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/boto/ec2/connection.py", line 1616, in request_spot_instances
    params['%s.UserData' % ls] = base64.b64encode(user_data)
  File "/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 1725, in <module>\n    main()\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 1719, in main\n    (tagged_instances, instance_dict_array, new_instance_ids, changed) = enforce_count(module, ec2, vpc)\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 963, in enforce_count\n    = create_instances(module, ec2, vpc, override_count=to_create)\n  File \"/var/folders/wz/__dd9trs0kl4jb3rs83704y80000gn/T/ansible_6sdx67__/ansible_module_ec2.py\", line 1253, in create_instances\n    res = ec2.request_spot_instances(spot_price, **params)\n  File \"/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/site-packages/boto/ec2/connection.py\", line 1616, in request_spot_instances\n    params['%s.UserData' % ls] = base64.b64encode(user_data)\n  File \"/Users/romanvg/.miniconda/envs/ansible24/lib/python3.6/base64.py\", line 58, in b64encode\n    encoded = binascii.b2a_base64(s, newline=False)\nTypeError: a bytes-like object is required, not 'str'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 0
}

PLAY RECAP *******************************************************************************************************************************************************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

/cc @vladsaveliev

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/amazon/ec2.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @lwade @tgerla @willthames
click here for bot help

[brainstorm]

Can anyone confirm this bug?

[abadger]

@brainstorm I don't use ec2 but I do fix Python3 bugs. Can you test whether this PR fixes your issue? #37628

[brainstorm]

Woah, thanks @abadger! Will give it a try ;)