Either the YAML parser or shell module insert leading whitespace on non-first lines

[garthk]

ISSUE TYPE

Bug Report

COMPONENT NAME

core

ANSIBLE VERSION

2.1

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

Demo:

• Copy the following to leading-whitespace-demo.yaml
• ansible-playbook -vi localhost, leading-whitespace-demo.yaml

#!/usr/bin/env ansible-playbook -vi localhost,
# leading-whitespace-demo.yaml

---
- name: demonstrate leading whitespace
  hosts: localhost
  connection: local
  gather_facts: no
  sudo: no

  tasks:

  - name: attempt
    shell: |
      echo first
      echo second

Output:

PLAY [demonstrate leading whitespace] ***************************************** 

TASK: [attempt] *************************************************************** 
changed: [localhost] => {"changed": true, "cmd": "echo first\n echo second", "delta": "0:00:00.005381", "end": "2015-08-21 16:06:50.534878", "rc": 0, "start": "2015-08-21 16:06:50.529497", "stderr": "", "stdout": "first\nsecond", "warnings": []}

PLAY RECAP ******************************************************************** 
localhost                  : ok=1    changed=1    unreachable=0    failed=0   

Note cmd is "echo first\n echo second": a space has been inserted at the beginning of the second line.

Run the same YAML through yaml.load, and there's no extra space:

$ python
Python 2.7.10 (default, Jul 14 2015, 19:46:27) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pprint, yaml
>>> pprint.pprint(yaml.load(file('leading-whitespace-demo.yaml').read()))
[{'connection': 'local',
  'gather_facts': False,
  'hosts': 'localhost',
  'name': 'demonstrate leading whitespace',
  'sudo': False,
  'tasks': [{'name': 'attempt', 'shell': 'echo first\necho second\n'}]}]
>>> ^D

Reproduced on Mac OS X and CentOS with Ansible 1.9.2.

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[chrrrles]

This behavior is not from shell, but from AnsibleModule's run_command method which quotes and sanitizes command arguments as a list before joining them together again as a space delimited string.

Removing P3 tag pending further discussion as to whether this is expected behavior for the shell module as it was not designed for batching multiple commands. This behavior is also exhibited by the raw module; for where it is may be more arguable label it as a bug:

[user@foo 12034]$ ansible-playbook -vvvi localhost, leading-whitespace-demo-with-raw.yaml                                                                                              
No config file found; using defaults
1 plays in leading-whitespace-demo-with-raw.yaml

PLAY [demonstrate leading whitespace] ******************************************

TASK [attempt] *****************************************************************
ESTABLISH LOCAL CONNECTION FOR USER: user
localhost EXEC echo first
 echo second
ok: [localhost] => {"changed": false, "rc": 0, "stderr": "", "stdout": "first\nsecond\n", "stdout_lines": ["first", "second"]}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0   

[bugchecker]

I vote to allow multiline.

Some languages (like python) cannot be reduced to single line form:

- name: Ensure that tokens are NFC-normalized
  shell: |
    from unicodedata import normalize
    if "{{ item }}" != normalize("NFC", "{{ item }}"):
      raise RuntimeError("Token {{ item }} isn't in NFC form")
  args:
    executable: python3
  with_items: ["être", "мёд", "mädchen"]
  delegate_to: localhost

Multiline scripts can keep atomic (perform a single action):

- name: Wait until the socket opens
  shell: |
    import socket, time
    from contextlib import closing, suppress
    connected = False
    with closing(socket.socket()) as sock:
      while not connected:
        with suppress(ConnectionAbortedError, ConnectionRefusedError):
          sock.connect(('localhost', 8080))
          connected = True
        time.sleep(.5)
  args:
    executable: python3

Also difference between command and shell notations may be more clearly.

Python example:

subprocess.run(["ls", "."])
subprocess.run("ls .", shell=True)

Dockerfile example:

CMD ["ls", "."]
CMD ls .

Ansible example (how could it be):

- command: ["ls", "."]
- command: 
    - ls
    - .
- shell: ls .

[rgm3]

This breaks here document use in the shell module. For example:

- name: set some network iface file properties with ansible
  shell: |
    augtool -LA -t 'Shellvars incl {{ifcfg_dir}}/ifcfg-*' <<EOAUG
    set /files{{ifcfg_dir}}/ifcfg-{{item}}/ONBOOT "yes"
    set /files{{ifcfg_dir}}/ifcfg-{{item}}/BOOTPROTO "none"
    set /files{{ifcfg_dir}}/ifcfg-{{item}}/NM_CONTROLLED "no"
    save
    EOAUG
  register: aug
  changed_when: aug.stdout|search("Saved")
  with_items: "{{ ansible_interfaces|difference(['lo']) }}"

The end-of-document marker will never be hit because a space is inserted before.

[abadger]

It appears that we already discussed this in a parallel issue/PR and decided the recommended practice should be to use the script module instead. That resulted in us adding a note to use the script module to the docs: #32863

We'll discuss this in the meeting and if consensus hasn't changed, close this issue then.

[quixoten]

That resulted in us adding a note to use the script module to the docs

Does the script module interpolate jinja as in @rgm3 example above? I tried converting my heredoc script to using the script module, but it doesn't look like it's parsing jinja. If the script module doesn't interpolate jinja, then getting the same functionality requires the template module, the shell module, and the file module (create script, run script, cleanup script). In previous ansible versions, this only required the shell module.

[phemmer]

Just hit this myself after upgrading from ansible 1.9. The proposal of using script is obnoxious as you now have to use multiple files (plus the already-mentioned template issue).

However it appears there is a really simple workaround. Just put the content in a variable instead. For some reason it seems the leading newline is only added when you assign directly to the script value.

- hosts: localhost
  tasks:
    - shell: |
        echo first
        echo second

    - shell: '{{content}}'
      vars:
        content: |
          echo first
          echo second

TASK [shell] **************************************************************************************************
changed: [localhost] => {"changed": true, "cmd": "echo first\n echo second", "delta": "0:00:00.002009", "end": "2019-01-12 01:10:27.909195", "rc": 0, "start": "2019-01-12 01:10:27.907186", "stderr": "", "stderr_lines": [], "stdout": "first\nsecond", "stdout_lines": ["first", "second"]}

TASK [shell] **************************************************************************************************
changed: [localhost] => {"changed": true, "cmd": "echo first\necho second\n", "delta": "0:00:00.002110", "end": "2019-01-12 01:10:28.059276", "rc": 0, "start": "2019-01-12 01:10:28.057166", "stderr": "", "stderr_lines": [], "stdout": "first\nsecond", "stdout_lines": ["first", "second"]}

[samdoran]

This is resolved in newer Ansible versions.

[bcoca]

    "changed": true,
    "cmd": "echo first\necho second\n",
    "delta": "0:00:00.002033",
    "end": "2021-05-07 11:12:06.966239",
    "invocation": {
        "module_args": {
            "_raw_params": "echo first\necho second\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "msg": "",
    "rc": 0,
    "start": "2021-05-07 11:12:06.964206",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "first\nsecond",
    "stdout_lines": [
        "first",
        "second"
    ]
}```