Multiline secrets are not correctly parsed

[djuarezg]

Bug:

What happened:
If you follow https://github.com/openshift/ansible-service-broker/blob/master/docs/secrets.md and try to add a multiline secret as in:

---
apiVersion: v1
kind: Secret
metadata:
    name: test
    namespace: openshift-automation-service-broker
stringData:
    "test1": "test1"
    "test2": "test2"
    "test_multiline": |-
      -----BEGIN RSA PRIVATE KEY-----
      <FIRST LINE OF THE SSH KEY>
      <SECOND LINE OF THE SSH KEY>

the Ansible Playbook Bundle will see an error while loading the secrets YAML file, as if it was using newlines to separate secrets:

ERROR! Syntax Error while loading YAML.
  could not find expected ':'
The error appears to have been in '/tmp/secrets': line 6, column 1, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
<FIRST LINE OF THE SSH KEY>
<SECOND LINE OF THE SSH KEY>
^ here

This happens as well if you use the base64 data secret.

What you expected to happen:

The secret should keep the newlines and be used as a parameter on the APB.

[djuarezg]

Mounted secrets are copied to /tmp/secrets so they can be passed as parameters to the playbook, but instead of producing this expected secrets file:

---
ACCESS_KEY: blah
SECRET_KEY: blah
SWARM_CLUSTER_KEYPAIR: |-
-----BEGIN RSA PRIVATE KEY-----
 blah
 blah
 blah
 -----END RSA PRIVATE KEY-----
openstack_admin__user: blah
openstack_admin_password: blah

They produce something like this, which will fail during parsing:

---
ACCESS_KEY: blah
SECRET_KEY: blah
SWARM_CLUSTER_KEYPAIR: -----BEGIN RSA PRIVATE KEY-----
 blah1 blah2
 blah3 ...
 -----END RSA PRIVATE KEY-----
openstack_admin__user: blah
openstack_admin_password: blah

[jmrodri]

The python tool used to create the secret does indeed use a newline for the data. I will fix this.

[djuarezg]

@jmrodri I saw that 1.4 is out, but is this bug fixed?

[jmrodri]

@djuarezg a pre-release of 1.4 is out. There is still quite a bit of time to get 1.4 bugs fixed. We don't have a good version mechanism for Release Candidates or betas.

[jmrodri]

Created bugzilla to track this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1649075

[jmrodri]

@djuarezg FINALLY looking into this. I can't seem to recreate the scenario you've mentioned above. I took the snippet of the tool that creates the secret to try to debug it. My secret ends up being base64 encoded for the values.

additional_keys.yml

---
ACCESS_KEY: blah
SECRET_KEY: blah
SWARM_CLUSTER_KEYPAIR: |-
 -----BEGIN RSA PRIVATE KEY-----
 blah
 blah
 blah
 -----END RSA PRIVATE KEY-----
openstack_admin__user: blah
openstack_admin_password: blah

The secret in openshift looks like this:

apiVersion: v1
data:
  ACCESS_KEY: ImJsYWgi
  SECRET_KEY: ImJsYWgi
  SWARM_CLUSTER_KEYPAIR: Ii0tLS0tQkVHSU4gUlNBIFBSSVZBVEUgS0VZLS0tLS0KYmxhaApibGFoCmJsYWgKLS0tLS1FTkQgUlNBIFBSSVZBVEUgS0VZLS0tLS0i
  key1: ImhlbGxvIg==
  key2: IndvcmxkIg==
  openstack_admin__user: ImJsYWgi
  openstack_admin_password: ImJsYWgi
kind: Secret
metadata:
  creationTimestamp: 2019-02-11T21:23:54Z
  name: testname
  namespace: broker-ns
  resourceVersion: "3238"
  selfLink: /api/v1/namespaces/broker-ns/secrets/testname
  uid: 55c378a5-2e43-11e9-a6ca-64006a559cc9
type: Opaque

If I decode the SWARM_CLUSTER_KEYPAIR with base64 I see the following:

"-----BEGIN RSA PRIVATE KEY-----
blah
blah
blah
-----END RSA PRIVATE KEY-----"

[jmrodri]

This is the script I was using to test it is a subset of the create_broker_secret.py script.
https://gist.github.com/jmrodri/169be3a1fc2cb232df81604481f9a6b6

[jmrodri]

#! /usr/bin/env python

import sys
import base64
import subprocess

# Output some nicer errors if a user doesn't have the required packages
try:
    import yaml
except Exception:
    print("No yaml parsing modules installed, try: pip install pyyaml")
    sys.exit(1)

# Work around python2/3 input differences
try:
    input = raw_input
except NameError:
    pass

USAGE = """USAGE:
  {command} NAME NAMESPACE IMAGE [BROKER_NAME] [KEY=VALUE]* [@FILE]*

  NAME:         the name of the secret to create/replace
  NAMESPACE:    the target namespace of the secret. It should be the namespace of the broker for most usecases
  IMAGE:        the docker image you would like to associate with the secret
  BROKER_NAME:  the name of the k8s ServiceBroker resource. Defaults to ansible-service-broker
  KEY:          a key to create inside the secret. This cannot contain an "=" sign
  VALUE:        the value for the  KEY in the secret
  FILE:         a yaml loadable file containing key: value pairs. A file must begin with an "@" symbol to be loaded


EXAMPLE:
  {command} mysecret ansible-service-broker docker.io/ansibleplaybookbundle/hello-world-apb key1=hello key2=world @additional_keys.yml

"""

DATA_SEPARATOR = "\n    "

SECRET_TEMPLATE = """---
apiVersion: v1
kind: Secret
metadata:
    name: {name}
    namespace: {namespace}
data:
    {data}
"""


def main():
    name = sys.argv[1]
    namespace = sys.argv[2]
    apb = sys.argv[3]
    if '=' not in sys.argv[4] and '@' not in sys.argv[4]:
        broker_name = sys.argv[4]
        idx = 4
    else:
        broker_name = None
        idx = 3

    keyvalues = list(map(
        lambda x: x.split("=", 1),
        filter(lambda x: "=" in x, sys.argv[idx:])
    ))
    files = list(filter(lambda x: x.startswith("@"), sys.argv[idx:]))
    data = keyvalues + parse_files(files)

    create_secret(name, namespace, data)


def parse_files(files):
    params = []
    for file in files:
        file_name = file[1:]
        with open(file_name, 'r') as f:
            params.extend(yaml.load(f.read()).items())
    return params


def create_secret(name, namespace, data):
    encoded = [(quote(k), base64.b64encode(quote(v))) for (k, v) in data]
    secret = SECRET_TEMPLATE.format(
        name=name,
        namespace=namespace,
        data=DATA_SEPARATOR.join(map(": ".join, encoded))
    )

    with open('/tmp/{name}-secret'.format(name=name), 'w') as f:
        f.write(secret)

    print('oc create -f /tmp/{name}-secret'.format(name=name))

    print('Created secret: \n\n{}'.format(secret))


def quote(string):
    return '"{}"'.format(string)

if __name__ == '__main__':
    if len(sys.argv) < 5 or sys.argv[1] in ("-h", "--help"):
        print(USAGE.format(command=sys.argv[0]))
        sys.exit()

    try:
        main()
    except Exception:
        print("Invalid invocation")
        print(USAGE.format(command=sys.argv[0]))
        raise

[jmrodri]

Can you provide a the sample input file and exactly how you were invoking the create_broker_secret.py script? I might be missing something.

[jmrodri]

No feedback in 22 days and I was not able to recreate the problem.

[djuarezg]

Sorry I could not come back to this issue before. Thank you for your script for local secret generation, it is really useful.

@jmrodri Just one question, secrets are eventually stored in /opt/apb/env/passwords. How do I access these variables from my Ansible roles?

Extravars are accessible through env vars, but passwords are not. If I want to access them I have to manually add a task including this file as vars.