copy content output from json and a variable is not idempotent under py3

[bstpierre]

ISSUE TYPE

• Bug Report

COMPONENT NAME

copy_module

ANSIBLE VERSION

ansible 2.5.0 (devel 34206a0402) last updated 2018/01/08 18:52:56 (GMT +000)
  config file = None
  configured module search path = ['/home/ubuntu/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/ubuntu/ansible/lib/ansible
  executable location = /home/ubuntu/ansible/bin/ansible
  python version = 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]

also seen on:

ansible 2.4.2.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/brian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/brian/.virtualenvs/mypy/lib/python3.5/site-packages/ansible
  executable location = /home/brian/.virtualenvs/mypy/bin/ansible
  python version = 3.5.2 (default, Nov 17 2016, 17:05:23) [GCC 5.4.0 20160609]

CONFIGURATION

(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ ansible-config dump --only-changed
(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 

(no output -- no changes to ansible.cfg, this is a clean ubuntu 16.04 VM with python, pip, virtualenv, a clone of the ansible repo, and pip -r requirements.txt)

OS / ENVIRONMENT

ubuntu 16.04, managing itself

SUMMARY

When running under python 3, using copy.content with literal json and a variable in the input, the output varies from run to run (the dict keys are being reordered arbitrarily).

STEPS TO REPRODUCE

Run the playbook below with:

ansible-playbook -i localhost, playbook.yaml

---
- hosts: localhost
  connection: local
  gather_facts: false

  vars:
    my_var: a_value

  tasks:
    - copy:
        dest: ./output.txt
        content: |
          {
              "key1": "{{ my_var }}",
              "key2": "something else",
              "key3": "something different",
              "key4": "completely different",
              "key5": "1.2.3"
          }

Note in the example playbook:

1. This only seems to happen when using a variable (static content didn't show this behavior).
2. This only seems to happen when the input is json (yaml didn't show this behavior).

EXPECTED RESULTS

1. The provided input should be output in the same format (i.e. without having whitespace rearranged).
2. The provided input should not be reordered (i.e. dict keys should not "move around").
3. (As a corollary to 2) The playbook should yield changed=0 on the second run -- the output file should not be modified.

ACTUAL RESULTS

See example run below.

Note that:

1. The input has been reformatted onto a single line.
2. The keys are reordered from the input.
3. The file changes on every run.

(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ ansible-playbook -i localhost, playbook.yaml 

PLAY [localhost] *******************************************************************************************************************************************

TASK [copy] ************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *************************************************************************************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0   

(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ cat output.txt 
{"key1": "a_value", "key2": "something else", "key5": "1.2.3", "key3": "something different", "key4": "completely different"}(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 
(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 
(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ ansible-playbook -i localhost, playbook.yaml 

PLAY [localhost] *******************************************************************************************************************************************

TASK [copy] ************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *************************************************************************************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0   

(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ cat output.txt 
{"key5": "1.2.3", "key3": "something different", "key4": "completely different", "key1": "a_value", "key2": "something else"}(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 
(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 
(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ ansible-playbook -i localhost, playbook.yaml 

PLAY [localhost] *******************************************************************************************************************************************

TASK [copy] ************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP *************************************************************************************************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0   

(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ cat output.txt 
{"key1": "a_value", "key5": "1.2.3", "key4": "completely different", "key3": "something different", "key2": "something else"}(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 
(ansible-test) ubuntu@ubuntu-xenial:~/ansible$ 

[ansibot]

Files identified in the description:

• lib/ansible/modules/files/copy.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[sivel]

You have properly identified that if the string contains a variable, that it triggers a change in functionality.

To step you through the process:

1. If the value contains a variable, it is processed through Templar.template
   a. Templar.template is powered by jinja2, and historically only has the ability to return strings
   b. Because jinja2 has historically only had the ability to template strings, we do some magic to try and convert strings that look like python data structures to python data structures
   c. The string now becomes a python dictionary
2. The copy action plugin, if it receives a python list or dict, it will json encode the data
3. The json encoded data gets written to the file to be placed on the remote machine

Because dicts in python are unordered, and we don't supply sort_keys=True to json.dumps, that results in the order of the keys changing. This is also the reason to why the newlines are removed.

We have recently gotten functionality into jinja2 to return native data types, and are working to integrate this functionality into ansible, which will allow ansible to stop guessing the data should be a python datatype, and instead of leaving alone.

Instead of using the content parameter with the copy module, you should instead use the template module, and place your content with the variable in the template module to avoid munging.

If you have further questions please stop by IRC or the mailing list:

• IRC: #ansible on irc.freenode.net
• mailing list: https://groups.google.com/forum/#!forum/ansible-project