Multi-Line Environment Variables Validation Errors

[probably-not]

In most companies, security policies dictate that we don't store secrets within a simple Git repository, and instead, inject the data via env variables to any CI/CD process that runs and requires the secrets.

While decK allows env variable injection, the injection is done in a simple way using Go Templates, without any regards to formatting of the file format. Subsequently, if the secret is a multi-line secret (such as a private or public RSA key necessary in certain plugins), then this breaks the file formatting and will cause errors in the file validation.

As an example, with a state file that looks like this:

_format_version: "1.1"
consumers:
  - custom_id: consumer_1
    jwt_secrets:
      - algorithm: RS256
        key: consumer_1_iss_key
        rsa_public_key: ${{ env "DECK_RSA_PUBLIC_KEY" }}
        secret: consumer_1_secret
    username: consumer_1_user_name

Running decK validate fails with the following error:

$ DECK_RSA_PUBLIC_KEY=$(cat pubkey.pem) deck validate

Error: reading file: validating file content: unmarshaling file content: error converting YAML to JSON: yaml: line 9: could not find expected ':'
exit status 1

[hbagdi]

I agree that the injection is a bit simplistic and we should be doing something more.
Very open to new ideas on making this better.

For the specific case, something like this may get work:

_format_version: "1.1"
consumers:
  - custom_id: consumer_1
    jwt_secrets:
      - algorithm: RS256
        key: consumer_1_iss_key
        rsa_public_key: |
          ${{ env "DECK_RSA_PUBLIC_KEY" }}
        secret: consumer_1_secret
    username: consumer_1_user_name

[probably-not]

For the specific case, something like this may get work:

Unfortunately that still fails with the same error, since the template is still injecting multiple lines into the file and the subsequent lines aren't indented.

Very open to new ideas on making this better.

@hbagdi What do you think about first unmarshaling the YAML file into a temporary map immediately after reading in the readContent function, and then iterating over the keys and making any replacements you need while iterating? Once you complete the replacements you could then use something like mapstructure.Decode to decode from a map to a Content struct.

[probably-not]

Either that, or the template funcs that are passed in need to be a lot more powerful so that they can enable the yaml (and JSON) formatting necessary

[hbagdi]

Either that, or the template funcs that are passed in need to be a lot more powerful so that they can enable the yaml (and JSON) formatting necessary

That is my current stance since I expect to not have a lot of such cases.
We use a single "env" function in the file today but the hopes and plans are bigger - having injections via other mechanisms. This would be a step in that direction.

[probably-not]

The one downside that I do know of to doing things raw within the template is that it requires a lot of extra overhead... for example, in this case, you not only need to "yaml-ize" the input, but also to ensure that it is indented the correct indentation level, and ensure that each line (since it is a multi-line input) is indented the correct indentation level as well.

It's adds some complexity to know the levels and to know how it will be rendered... which is made even more complicated because of the fact that there's no render command to only render what the actual config will look like pre-validation...

[GGabriele]

A workaround for this would be to use the JSON format for the configuration file, instead of YAML (please note the backticks around the variable name):

$ cat kong.json
{
  "_format_version": "1.1",
  "consumers": [
    {
      "custom_id": "consumer_1",
      "username": "consumer_1_user_name",
      "jwt_secrets": [
        {
          "algorithm": "RS256",
          "key": "consumer_1_iss_key",
          "rsa_public_key": "${{ env `DECK_RSA_PUBLIC_KEY` }}",
          "secret": "consumer_1_secret"
        }
      ]
    }
  ]
}

$ deck sync -s kong.json
creating consumer consumer_1_user_name
creating jwt-auth consumer_1_iss_key for consumer 73a0b0ef-9f75-4d91-8fbb-093a8ca72fac
Summary:
  Created: 2
  Updated: 0
  Deleted: 0

[probably-not]

@GGabriele I just tried running with the JSON format, it caused a schema violation error.

$ cat kong.json
{
  "_format_version": "1.1",
  "consumers": [
    {
      "custom_id": "consumer_1",
      "username": "consumer_1_user_name",
      "jwt_secrets": [
        {
          "algorithm": "RS256",
          "key": "consumer_1_iss_key",
          "rsa_public_key": "${{ env `DECK_RSA_PUBLIC_KEY` }}",
          "secret": "consumer_1_secret"
        }
      ]
    }
  ]
}

```shell
$ DECK_RSA_PUBLIC_KEY=$(cat ./auth0_rsa_public.key) deck --state "./kong.json" sync
creating consumer consumer_1_user_name
creating jwt-auth consumer_1_iss_key for consumer faeb4f73-45aa-45b5-aa33-d1b10fd4bc9a
Summary:
  Created: 1
  Updated: 0
  Deleted: 0
Error: 1 errors occurred:
        while processing event: {Create} jwt-auth consumer_1_iss_key for consumer faeb4f73-45aa-45b5-aa33-d1b10fd4bc9a failed: HTTP status 400 (message: "2 schema violations (failed conditional validation given value of field 'algorithm'; rsa_public_key: invalid key)")

[GGabriele]

@probably-not I see what's wrong, my env variable was well formatted but getting it directly from a file is more problematic. Can you try something like this?

$ DECK_RSA_PUBLIC_KEY=$(cat ./auth0_rsa_public.key | jq -sR . | tr -d '"' ) deck --state "./kong.json" sync
Summary:
  Created: 0
  Updated: 0
  Deleted: 0

[dyeske61283]

The workaround, with using the json format on the state files, is working in my case. Thank you!

[probably-not]

Good to hear! I'll try it out in my pipeline to see if it works well.

[GGabriele]

Thank you both for the patience on this!

[probably-not]

@GGabriele Looks like this works in our pipeline, the only issue I found is that the tools (jq, tr) are not available in the base deck Docker image, so we had to create our own image and add in tools manually. If we can add these tools to the deck Dockerfile that would be a big help.

[GGabriele]

@probably-not I just tried with the latest deck image and it seems that tr tool is already part of it:

/ $ deck version
decK kong/deck:v1.12.0 (e6da44844f150ea3d15dfffe00bde471d96bbb4e)

/ $ jq
/bin/sh: jq: not found

/ $ tr
BusyBox v1.34.1 (2022-04-04 10:19:27 UTC) multi-call binary.

Usage: tr [-cds] STRING1 [STRING2]

Translate, squeeze, or delete characters from stdin, writing to stdout

	-c	Take complement of STRING1
	-d	Delete input characters coded STRING1
	-s	Squeeze multiple output characters of STRING2 into one character

Just opened a PR to add jq though: #660

Thanks for confirming this works well with you!

[probably-not]

Excellent, thanks 👍

[ffjeremy]

For anyone else coming here having the same problems I had (passing a multi-line public key into the declarative config as an environment variable), I was able to make it work in YAML like this:

Replace newlines with "\n" in the public key and assign that to the environment variable:

export DECK_RSA_PUBKEY="-----BEGIN PUBLIC KEY-----\n..."

And then in the declarative config YAML:

rsa_public_key: "${{ env `DECK_RSA_PUBKEY` }}"

The above usage takes advantage of the fact that double quotes in YAML convert "\n" to newlines. I'm also using the backticks for quoting the environment variable name -- apparently this is a Go Template thing.

Anyway... there you have it. Worked for me.