Creating Docker secrets from file system

[sgtpanic]

ISSUE TYPE

• Feature Idea

COMPONENT NAME

docker_secret

ANSIBLE VERSION

ansible 2.4.0.0
  config file = None
  configured module search path = [u'/Users/me/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.4.0.0/libexec/lib/python2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.14 (default, Sep 25 2017, 09:54:19) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)]

CONFIGURATION

NA

OS / ENVIRONMENT

N/A

SUMMARY

I think that docker_secret is an excellent addition to Ansible and I am interested to use it to manage infrastructure. I looked at it today but it appears that I can only create secrets from strings, not directly from files (e.g. I have a property file that I would like to turn into a secret or convert an SSL private key into a secret). I considered cating the file into a variable but I was hoping for something more direct. Is this correct and I'm limited to only strings? I understand it's only in preview so it's not the final iteration. If I can't create secrets from files, I'd be glad to add functionality to the module with some guidance.

STEPS TO REPRODUCE

I was hoping to do something like the below.

name: create a new docker secret
docker_secret:
  name: my_new_secret
  data: /path/to/secret
  file: yes
  state: present

EXPECTED RESULTS

ACTUAL RESULTS

[ansibot]

Files identified in the description:

• lib/ansible/modules/cloud/docker/docker_secret.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

[ansibot]

cc @chouseknecht
click here for bot help

[sivel]

This can be accomplished by using "{{ lookup('file', '/path/to/file') }}" which mimics the same functionality that other modules recommend.

I'll however leave this open as a feature idea as there may be some reason why lookup won't suffice, where the file you are trying to use has non ascii characters that could cause trouble.

[sgtpanic]

@sivel Thank you for the info. I'll try that pattern and I expect that it should work for me.

[sgtpanic]

I was able to use lookup for nearly everything but I do have JKSs that I would like to add as secrets and they are binary files. I tried to add them earlier today but was unsuccessful. I'm willing to contribute an update if it is something that the project would like to add. @sivel do you have any advice? Thanks again.

[ssullivan]

@sivel Will lookup modify the data it reads in? I've seen some weird behavior that I'm tracking down where JSON loaded through this method appears to be modified when viewing it inside the running service container.

Here are some of the changes that I've observed so far:

• newlines are removed
• true is changed to True
• false is changed to False
• null is changed to None

[sivel]

The modification of JSON data is likely very closely related to the info provided in #34595 (comment)

Trying to pipe json through jinja2 in ansible currently, will end up converting the JSON to a python dictionary.

[ansibot]

cc @felixfontein
click here for bot help

[ansibot]

cc @akshay196 @danihodovic @dariko @jwitko @kassiansun @tbouvet
click here for bot help

[felixfontein]

@sgtpanic I think it would be good to have the option to create secrets from binary data (which currently doesn't really work). There are two options:

• add an option to docker_secret to base64-decode the received data; then one can use the lookup filter combined with the b64encode filter to safely transport binary data to the docker_secret module;
• add an option to read the secret from a file.

Both options have advantages and disadvantages:

• file lookup only allows to fetch something from the local machine, not from a remote machine;
• if docker_secret can read the secret from a file, the file must be on the remove machine.
  Depending on your threat model, you might not want to move the secret into a file on the remote machine (i.e. you can't use the file option), or you don't want to first pull the secret from the remote machine to the master (i.e. you cannot use docker_secret module if it doesn't have a file option).

@ushuz since you added the docker_config module, which is very similar to the docker_secret module: do you have any opinions/ideas/... on this? Something similar might also be useful for docker_config, so you can put binary data into docker config objects.

[ushuz]

@felixfontein

For docker_secret and docker_config, I don't think there are strong need for reading a file on remote, nor it's a good practice in general. Even if there are legit use cases, one can always use slurp module. While reading file from control machine is well covered by lookup filter, there's no reason to reinvent the wheel inside each module.

However, if the module itself or lookup somehow can not handle binary input very well, like in @sgtpanic's case probably, we should fix that instead. lookup certainly messed up JSON input for sure.

---

I tried to add them earlier today but was unsuccessful.

@sgtpanic Could you provide more details? Is it lookup or docker_secret that caused the failure? You can rerun the playbook with -vvv or even more v to see more verbose output.

[felixfontein]

@ushuz That JSON didn't really work is not that much a surprise, since Ansible tries (and has to try, for a lot of things to work!) at different places to interpret strings as YAML (which is a super-set of JSON), and re-encodes them when a string is needed. That's why keeping stuff base64 encoded is a good thing (as Ansible then won't modify it). It helps both for JSON and for binary data :)

I had some trouble with binary data when playing with TLS ALPN authentication for ACME (there, the ACME server returns a binary blob which needs to be inserted into a x509 certificate); it didn't work very well (there were some less/more problems with Python 2 or 3, I don't remember which), and I asked on the ansible-devel mailing list how to handle binary data, and it was suggested to move it around base64 encoded. (That's also how the slurp module returns it.)

I guess the easiest will be to add an option to docker_secret and docker_config which allows to base64-decode data before using it. With a default value of false it will not break backwards compatibility as well :)

(And for lookup: if one combines lookup with b64encode, like data: "{{ lookup('file', '/path/to/file') | b64encode }}", everything should work well with such an option -- both for JSON and binary input.)

[ushuz]

@felixfontein

The option you proposed seems applicable to any modules that potentially accepts binary data. As the root causes are some Ansible design decisions and Python 2 / 3, IMO it would be better to handle this at a higher level, instead of dealing it in each module affected.

For JSON, lookup() | string is an acceptable workaround I found here: #7005 (comment). Maybe binary data could use a similar approach? And of course better documentation.

[felixfontein]

@ushuz that workaround is great, it actually helped alot somewhere else: #48984 If it helps here, great! :)

It won't help for binary data, though. There's too much conversion to/from strings which kills it.

[felixfontein]

I created a PR for this (#49688), which adds an option to Base64-decode data. It's both for docker_secret and docker_config.

[ansibot]

cc @DBendit
click here for bot help

[felixfontein]

With the PR just merged (which will be included in Ansible 2.8, to be released this spring/summer), you can do

docker_secret:
    data: "{{ lookup('file', '/path/to/secret/file') | base64 }}"
    data_is_b64: true
    ...

This only works for files on the control node; for files on the remote node, you need to slurp them first (which will give you a base64 encoded string, i.e. you can also use the new data_is_b64 option).