New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add vars plugin for vars files encrypted with Mozilla's sops #38843

Open
wants to merge 9 commits into
base: devel
from

Conversation

Projects
None yet
5 participants
@schlueter
Contributor

schlueter commented Apr 16, 2018

SUMMARY

This adds a vars plugin for loading decrypted vars from vars files which were encrypted with Mozilla's sops.

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

Sops vars plugin.

ANSIBLE VERSION
ansible 2.6.0
  config file = /Users/bschlueter/workspace/neptune/ansible.cfg
  configured module search path = ['/usr/share/ansible']
  ansible python module location = /Users/bschlueter/.config/zsh/pyenv/versions/3.6.4/lib/python3.6/site-packages/ansible
  executable location = /Users/bschlueter/.config/zsh/pyenv/versions/3.6.4/bin/ansible
  python version = 3.6.4 (default, Feb 15 2018, 12:21:02) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)]
ADDITIONAL INFORMATION

Sops is advantageous over Ansible Vault because it allows encryption of values with pgp or kms rather than just passwords.

Currently this only works with pgp keys, not kms. I'm working on getting kms working, but wanted to get this out there for eyeballs.

@ansibot

This comment has been minimized.

Contributor

ansibot commented Apr 16, 2018

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:337:0: anomalous-backslash-in-string Anomalous backslash in string: '\['. String constant might be missing an r prefix.
lib/ansible/plugins/vars/sops.py:340:0: anomalous-backslash-in-string Anomalous backslash in string: '\]'. String constant might be missing an r prefix.

The test ansible-test sanity --test boilerplate [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:0:0: missing: __metaclass__ = type
lib/ansible/plugins/vars/sops.py:0:0: missing: from __future__ import (absolute_import, division, print_function)

click here for bot help

@ansibot

This comment has been minimized.

Contributor

ansibot commented Apr 17, 2018

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:338:0: anomalous-backslash-in-string Anomalous backslash in string: '\['. String constant might be missing an r prefix.
lib/ansible/plugins/vars/sops.py:341:0: anomalous-backslash-in-string Anomalous backslash in string: '\]'. String constant might be missing an r prefix.

click here for bot help

@ansibot ansibot added the stale_ci label Apr 25, 2018

@mattclay

This comment has been minimized.

Member

mattclay commented Apr 26, 2018

CI failure in Python 2.x unit tests: https://app.shippable.com/github/ansible/ansible/runs/61641/3/tests

CI failure in integration tests due to traceback:

2018-04-17 13:51:55 Traceback (most recent call last):
2018-04-17 13:51:55   File "/root/ansible/bin/ansible-playbook", line 118, in <module>
2018-04-17 13:51:55     exit_code = cli.run()
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/cli/playbook.py", line 122, in run
2018-04-17 13:51:55     results = pbex.run()
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/executor/playbook_executor.py", line 159, in run
2018-04-17 13:51:55     result = self._tqm.run(play=play)
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/executor/task_queue_manager.py", line 289, in run
2018-04-17 13:51:55     play_return = strategy.run(iterator, play_context)
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/plugins/strategy/linear.py", line 248, in run
2018-04-17 13:51:55     task_vars = self._variable_manager.get_vars(play=iterator._play, host=host, task=task)
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/vars/manager.py", line 295, in get_vars
2018-04-17 13:51:55     all_vars = combine_vars(all_vars, locals()[entry]())
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/vars/manager.py", line 262, in all_plugins_inventory
2018-04-17 13:51:55     return _plugins_inventory([all_group])
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/vars/manager.py", line 243, in _plugins_inventory
2018-04-17 13:51:55     for plugin in vars_loader.all():
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/plugins/loader.py", line 489, in all
2018-04-17 13:51:55     module = self._load_module_source(name, path)
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/plugins/loader.py", line 357, in _load_module_source
2018-04-17 13:51:55     module = imp.load_source(full_name, path, module_file)
2018-04-17 13:51:55   File "/root/ansible/lib/ansible/plugins/vars/sops.py", line 9
2018-04-17 13:51:55 SyntaxError: Non-ASCII character '\xc3' in file /root/ansible/lib/ansible/plugins/vars/sops.py on line 9, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
@schlueter

This comment has been minimized.

Contributor

schlueter commented May 3, 2018

Ah, my coworker noticed that the encoding needs to be stated in the python file, updating with that.

@schlueter schlueter force-pushed the schlueter:feature/sops-vars-plugin branch 2 times, most recently May 3, 2018

@ansibot

This comment has been minimized.

Contributor

ansibot commented May 3, 2018

The test ansible-test sanity --test boilerplate [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:0:0: missing: __metaclass__ = type
lib/ansible/plugins/vars/sops.py:0:0: missing: from __future__ import (absolute_import, division, print_function)

The test ansible-test sanity --test compile --python 2.6 [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:266:53: SyntaxError: def _walk_and_decrypt(self, branch, key, aad=rb'', stash=None, digest=None,

The test ansible-test sanity --test compile --python 2.7 [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:266:53: SyntaxError: def _walk_and_decrypt(self, branch, key, aad=rb'', stash=None, digest=None,

click here for bot help

@ansibot

This comment has been minimized.

Contributor

ansibot commented May 3, 2018

The test ansible-test sanity --test boilerplate [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:0:0: missing: __metaclass__ = type
lib/ansible/plugins/vars/sops.py:0:0: missing: from __future__ import (absolute_import, division, print_function)

The test ansible-test sanity --test compile --python 2.6 [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:266:53: SyntaxError: def _walk_and_decrypt(self, branch, key, aad=rb'', stash=None, digest=None,

The test ansible-test sanity --test compile --python 2.7 [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:266:53: SyntaxError: def _walk_and_decrypt(self, branch, key, aad=rb'', stash=None, digest=None,

click here for bot help

@ansibot

This comment was marked as resolved.

Contributor

ansibot commented May 4, 2018

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/modules/cloud/ovirt/ovirt_permissions_facts.py:97:47: ansible-format-automatic-specification Format string contains automatic field numbering specification
lib/ansible/modules/cloud/ovirt/ovirt_permissions_facts.py:98:20: ansible-format-automatic-specification Format string contains automatic field numbering specification

The test ansible-test sanity --test pep8 [explain] failed with 3 errors:

lib/ansible/modules/cloud/ovirt/ovirt_permissions_facts.py:98:21: E126 continuation line over-indented for hanging indent
lib/ansible/modules/cloud/ovirt/ovirt_permissions_facts.py:99:17: E126 continuation line over-indented for hanging indent
lib/ansible/modules/cloud/ovirt/ovirt_permissions_facts.py:100:13: E123 closing bracket does not match indentation of opening bracket's line

click here for bot help

@mattclay

This comment has been minimized.

Member

mattclay commented May 4, 2018

The unrelated errors have been fixed. I've restarted CI for this PR.

lib/ansible/plugins/vars/sops.py Outdated
from base64 import b64decode
from socket import gethostname
import boto3

This comment has been minimized.

@mattclay

mattclay May 4, 2018

Member

CI failure due to no try/except ImportError block around import from outside of the Python standard library.

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 5, 2018

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:40:10: undefined-variable Undefined variable 'AnsibleError'
lib/ansible/plugins/vars/sops.py:41:0: trailing-whitespace Trailing whitespace

The test ansible-test sanity --test pep8 [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:41:1: W293 blank line contains whitespace

click here for bot help

@mattclay

This comment has been minimized.

Member

mattclay commented Jun 12, 2018

CI failure in integration tests due to a traceback:

2018-06-08 18:49:23 Traceback (most recent call last):
2018-06-08 18:49:23   File "/root/ansible/bin/ansible-playbook", line 118, in <module>
2018-06-08 18:49:23     exit_code = cli.run()
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/cli/playbook.py", line 122, in run
2018-06-08 18:49:23     results = pbex.run()
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/executor/playbook_executor.py", line 159, in run
2018-06-08 18:49:23     result = self._tqm.run(play=play)
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/executor/task_queue_manager.py", line 289, in run
2018-06-08 18:49:23     play_return = strategy.run(iterator, play_context)
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/plugins/strategy/linear.py", line 282, in run
2018-06-08 18:49:23     task_vars = self._variable_manager.get_vars(play=iterator._play, host=host, task=task)
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/vars/manager.py", line 295, in get_vars
2018-06-08 18:49:23     all_vars = combine_vars(all_vars, locals()[entry]())
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/vars/manager.py", line 262, in all_plugins_inventory
2018-06-08 18:49:23     return _plugins_inventory([all_group])
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/vars/manager.py", line 243, in _plugins_inventory
2018-06-08 18:49:23     for plugin in vars_loader.all():
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/plugins/loader.py", line 489, in all
2018-06-08 18:49:23     module = self._load_module_source(name, path)
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/plugins/loader.py", line 357, in _load_module_source
2018-06-08 18:49:23     module = imp.load_source(full_name, path, module_file)
2018-06-08 18:49:23   File "/root/ansible/lib/ansible/plugins/vars/sops.py", line 40, in <module>
2018-06-08 18:49:23     raise AnsibleError("The vars plugin sops requires boto3.")
2018-06-08 18:49:23 NameError: name 'AnsibleError' is not defined
@ansibot

This comment has been minimized.

Contributor

ansibot commented Jun 12, 2018

The test ansible-test sanity --test pylint [explain] failed with 2 errors:

lib/ansible/plugins/vars/sops.py:40:10: undefined-variable Undefined variable 'AnsibleError'
lib/ansible/plugins/vars/sops.py:41:0: trailing-whitespace Trailing whitespace

The test ansible-test sanity --test pep8 [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:41:1: W293 blank line contains whitespace

click here for bot help

@jimi-c

This comment has been minimized.

Member

jimi-c commented Jun 14, 2018

Hi @schlueter, unfortunately any plugins contributed to Ansible need to be GPLv3+ licensed. If you can correct that we will continue considering this for inclusion.

lib/ansible/plugins/vars/sops.py Outdated
@@ -0,0 +1,464 @@
# -*- coding: utf-8 -*-
# This Source Code Form is subject to the terms of the Mozilla Public

This comment has been minimized.

@jimi-c

jimi-c Jun 14, 2018

Member

Plugins contributed to Ansible must be GPLv3+ licensed.

This comment has been minimized.

@schlueter

schlueter Jul 13, 2018

Contributor

Okay, I believe I can do that. Since this is heavily borrowed from https://github.com/mozilla/sops/blob/python-sops/sops/__init__.py, fyi @jvehent (the last contributor to that file) that I will be submitting this under the GPLv3+ rather than the MPL v2.

@schlueter schlueter force-pushed the schlueter:feature/sops-vars-plugin branch to 1470db6 Jul 17, 2018

@ansibot

This comment has been minimized.

Contributor

ansibot commented Jul 17, 2018

The test ansible-test sanity --test pylint [explain] failed with 1 error:

lib/ansible/plugins/vars/sops.py:53:10: undefined-variable Undefined variable 'AnsibleError'

click here for bot help

@mattclay

This comment has been minimized.

Member

mattclay commented Jul 17, 2018

In addition to the sanity test failures, there are failures in integration tests. A few of those:

https://app.shippable.com/github/ansible/ansible/runs/74458/11/console

ERROR! coercing to Unicode: need string or buffer, NoneType found

https://app.shippable.com/github/ansible/ansible/runs/74458/69/console

ERROR! expected str, bytes or os.PathLike object, not NoneType

from socket import gethostname
try:
import boto3

This comment has been minimized.

@mattclay

mattclay Jul 17, 2018

Member

Set HAS_BOTO3 to True on successful import and False on failure to import. Then check that value before using any boto3 imported code.

Raising an exception immediately on import failure prevents Ansible from working when boto3 is not available -- so make sure you test behavior of this plugin when boto3 is not available.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment