Ensure _raw_params retain exact spaces

[dagwieers]

SUMMARY

This fixes a longstanding bug related to multi-line YAML strings that lack the original whitespaces and gain a few new whitespaces after newlines.

ISSUE TYPE

• Bugfix Pull Request
• Docs Pull Request

COMPONENT NAME

shell (and others)

ANSIBLE VERSION

v2.8 and older

ADDITIONAL INFORMATION

This PR fixes issues with input handling. If you would do the following:

- hosts: localhost
  tasks:
  - shell: |
      ls
      cd
      echo "Foo"

This would translate to: ls\n cd\n echo "Foo"

Any indentation would get lost, all prefixed and trailing whitespaces are eaten and everything is joined with whitespaces (so one whitespace was always added after a newline).

This breaks various things link indentation, but most importantly here-documents in the shell module.

- hosts: localhost
  tasks:
  - shell: |
      cat <<EOF
        One
        Two
          Three
        Four
      EOF

This would translate into something that fails to work as designed:

cat <<EOF
 One
 Two
 Three
 Four
 EOF

[dagwieers]

Given the impact (this has bothered me for years) we may want to back-port this.

[alikins]

tests?

[dagwieers]

@alikins Sure, but apparently there's no consensus that we want to fix this.

[bcoca]

code looks fine, but i would add tests for this use case. Specially since string manipulation is very finicky across python versions.

Also i would leave the note about using script module, it just keeps playbooks more readable.

[abadger]

At today's meeting no one objected to this and several people expressed interest. However, the backwards compatibility was a question with mitigation for that being to have a config toggle and deprecation period. Both bcoca and I wanted the documentation to continue recommending script. Two reasons for that would be that it keeps playbooks more readable and the parsing code seems to be a never ending source of cornercases as it has to make decisions about reformatting which may not always be obvious or the way people want it. dag noted that he wasn't sure if docs should advise people to do something else as it might be that the usage fit their use case fine.

[dagwieers]

@bcoca @abadger Do you think backward compatibility breaks because of correct spacing ? I have tried to wrap my head around it, but I don't see how this could fail for existing use-case. Either the incorrect spacing was an issue (and people could not use the shell module) or it did work and correct spacing was irrelevant (and this has no real impact).

Can we maybe add it like this and see of people report any issues before adding a flag and deprecation ?

[bcoca]

I don't know about any case that it breaks existing playbooks, I would be surprised anyone found the current shell + multiline useful .. but who knows? people do weird stuff.

I would add to devel and let it macerate for a while, then consider backports.

[dagwieers]

I think I took in all feedback, please re-review.

I think I incorporated all feedback, can you re-review ?

[dagwieers]

rebuild_merge

[mattclay]

These changes result in a traceback for some inputs, see #46379 for details.