Copy module with base64-encoded binary file adds extra character

[ojkastl]

ISSUE TYPE

• Bug Report

COMPONENT NAME

copy module

ANSIBLE VERSION

Happens with both 2.2.0 and 2.2.1rc3

ansible 2.2.0.0
  config file = /Users/xyz/.ansible.cfg
  configured module search path = Default w/o overrides

CONFIGURATION

Nothing relevant: inventory, control_path, retry_files_enabled, ...

OS / ENVIRONMENT

Running ansible on OSX 10.12 and openSUSE Tumbleweed

SUMMARY

I generate a base64-encoded string from a file binary_file via
cat binary_file | base64 (or gbase64 in my case)
I put this string into a variable and use the copy module with content="..." to create a file. But the file differs from the original file, there is one additional character in the output.

Not sure if this is similar to #5262...

STEPS TO REPRODUCE

Create the binary file by e.g.
echo sxZARwIVokeqOMGPygc1S20CaGPiKDRGRzg0oSVGmCF2oXHua+9fVhriUQRd8vkmvpHoBmSsI6Y= | gbase64 -d >binary_file.2
Try this playbook.

- hosts: localhost
  vars:
    string_in_base64: sxZARwIVokeqOMGPygc1S20CaGPiKDRGRzg0oSVGmCF2oXHua+9fVhriUQRd8vkmvpHoBmSsI6Y=

  tasks:
    - copy:
        content: '{{ string_in_base64 | b64decode }}'
        dest: "/some/path/binary_file.ansible"

Compare the results with any checksum:
md5sum binary_file*

EXPECTED RESULTS

The created file and the original file should be identical.

ACTUAL RESULTS

The file is created and the content is almost right:

cat binary_file; echo ""; cat binary_file.ansible; echo ""

�@G�G�8��5Kmhc�(4FG84�%F�!v�q�k�_V�Q]��&���d�#�
�@G�G�8���5Kmhc�(4FG84�%F�!v�q�k�_V�Q]��&���d�#�

The second file actually has an additional character (at column 9).

[nitzmahone]

cc @abadger

At first glance, this smacks of text encoding issues somewhere between YAML/Jinja - I'm guessing it's probably a case of "don't do that", or extend copy to accept base64 natively via content_base64 or something to ensure that we don't trip over any encoding issues with everything else in the signal path...

[nitzmahone]

In the absence of a workaround, I'd suggest just including the file next to the playbook and using copy in its classical usage, rather than trying to embed the file content directly into the playbook...

[abadger]

yeah, content is only text compatible (utf-8 encoded text specifically.) trying to pass binary contents that way isn't going to work.

[ojkastl]

As it almost works, it would be nice if the copy module could be enhanced to accept binary data in base64, maybe some "base64_content: yes" switch?

In #5262 this was mentioned:
rstrip('\r\n')
Maybe this could be added if the base64_content switch is set to yes?

(Full disclosure, I have no clue of python, YAML and jinja, so it might seem easier for me than it is to implement that in real life...)

[ojkastl]

Also, something like this has been mentioned before, with a little workaround:

https://stackoverflow.com/questions/22773294/

The workaround is based on copying a text file to the target and then start the shell module to execute base64 on that file. Yes, thats ugly.

[ojkastl]

I also wanted to avoid having lots of encrypted files in my role. Having lots of variables in the existing host_vars seems to much nicer.

[bcoca]

copy can deal with binaries and encrypted files, the problem here is not copy but the templating system which is what you hit when using content, it was never designed to do this.

IF you are using vault to encrypt, copy will automatically decrypt on the fly.

Closing this as this is not a bug, it is just not designed to work this way.

[ojkastl]

What a shame. It was 'almost' working. But I guess you are right, this smells like abusing the templating...

[uncle-fed]

copy can deal with binaries and encrypted files,
the problem here is not copy but the templating system
which is what you hit when using content,
Closing this as this is not a bug, it is just not designed to work this way.

@bcoca : the explanation sort of makes sense (if you accept the fact that using 'content' equals using the templating system). However, if using "content" is not the right way to put some binary data from a variable into a file, then how else should it be done "properly" in ansible world (forgive my ignorance if this is possible but I did not find any other way)?

very simple use case: I have a tiny binary blob of data coming from some external database that ends up in an ansible variable (it gets there via one of the 'lookup' plugins) and now that binary data needs to be copied to a remote server and put into a file there.

How should this be done reliably, if "content" is not the right way and "was never designed for that"? Running shell commands that transform the data that might be base64 encoded is out of question (no idempotency), using module "command" with "creates" parameter is "kind of idempotent" but still ugly (it means we still need to convert binary data twice: first to base64 that maybe will pass as 'content', hopefully unchanged by it, and then from base64 again with the "command" module) - very strange extra conversion steps that should not be needed, really.

With regards to "not designed to work this way". Look at what the documentation states for copy / content parameter:

When used instead of src, sets the contents of a file directly to the specified value.
For anything advanced or with formatting also look at the template module.

"Directly to the specified value" (without any further explanations) kind of suggests that it should be binary safe. And to even further support this, it says "use template module, if you want formatting applied". So all together this reads as "this option just dumps stuff into a file "as is" and if you want more, dont use it, use templates instead". Of course, with such a tiny amount of explanation one can argue about how it is really supposed to work to infinity but the implementation is clear: it uses templating engine somehow.

Why should it act this way? Is there a good reason for it to even touch templating engine? If somebody wants templating features they should use templates. So how about making this feature binary safe?

[uncle-fed]

@bcoca : Alternative suggestion: if "content" intention is different from just dumping variable contents into a file "as is", can we have new option for the "copy" module? For instance, call it "data" instead of "content". That should be binary safe and not touch templating.