handling secrets with gitlabform

[weakcamel]

GitlabForm can be succesfully used to provision Gitlab projects with secrets, like passwords, API keys etc.

This is extremely helpful. It also means that with the current design, those secrets need to be hardcoded inside the config.yml file

It be great if sensitive data could be passed from external sources. Reasons:

• security; developers in the company may be privy to seeing general configuration of Gitlab so as a reference, so that they know what to expect in a specific project.. yet at the same time, secrets shouldn't be that easily readable
• single source of truth: users of GitlabForm may be keeping their secrets in various safe places and apps: Vault, LastPass, other password managers, even in an encrypted text file or a sticky note (in a safe, obviously :) )
• additionallly (nice to have), the secrets shouldn't be visible in plain text in GitlabForm logs

At the moment the only workaround for that that I can think of would be to keep have config.yml file as a template to be filled in at runtime (e.g. with https://pypi.org/project/j2cli/ or https://pypi.org/project/jinja2-vault/). This however:

• complicates the flow as you'd need to re-generate the config each time you want to apply it, lint it (see https://github.com/adrienverge/yamllint)
• may interfere with the current templating capabilities (re {{ project }} / {{ group }}).

Do you ( @gdubicki - project owner) have any plans (or even just opinions) on such functionality?

[gdubicki]

Hi @weakcamel , thank you for this report.

Frankly I have not thought about adding secure secrets handling to gitlabform yet.

At Egnyte we deploy gitlabform using Puppet, so we keep its config in a git repo, with all the values that are secret encrypted using asymmetric encryption - https://github.com/voxpupuli/hiera-eyaml. So on the machine which runs gitlabform once a day the config file is deployed in unencrypted form. This allows us to give developers access to the config repo and the public key so they can add MRs to add their new secrets encrypted. It is not perfect, but it's is safe enough for us thanks to other means of security that we use.

I think you can use hiera-eyaml even without using Puppet as it has cli to generate encryption keys, encrypt single value, edit encrypted file, decrypt single value, decrypt whole file - all the features to deploy the flow similar to ours. There is also https://github.com/mozilla/sops that seems to work in a similar way and has more features, but I have not use it myself.

Of course, as always, any PRs to gitlabform adding securing of the secrets into it are welcome too!

[weakcamel]

For your information (and for the record): for now I've worked around that with Jinja2 templates.
It's not quite ideal, however using vault-cli helped me to very easily fill in the secrets directly from Hashicorp Vault.

Once peopledoc/vault-cli#113 is finished, I'm going to also break down the config.yml into smaller bits as I expect we may end up with quite a sizeable configuration.

I may still look at adding secret support (or maybe Jinja2 support?) to Gitlabform at some point, but probably not just yet.

[Dridge]

I had the same problem, ended up just using sed in the gitlab-ci.yml to replace like so:
sed -i -e 's/SED_REPLACE_JIRA_PASSWORD/'"$JIRA_PASSWORD"'/g' config.yml
It will still end up in the log though 😢

[gdubicki]

How about adding support for automatic replacement of some strings with environment variables + changing logging code to not print any values with verbosity level up to „verbose” (they would still be printed with „debug” on though).

This would allow setting them as secrets in GitLab project, setting them from Vault in a pre-script etc. and seems easy to implement.

What do you think?

[gdubicki]

...or we could add built-in Jinja2 support to automatically pre-process the while config with built-in code similar to answers for https://stackoverflow.com/q/25862071/2693875 to be able to reference env variables anywhere.

This would generalize @mkjmdski ’s code for Jinja2 support for file contents and setting GitLab URL and token from env variables.

[gdubicki]

...and this could remove uglyness of group, project and usernames used in the integration tests being both hardcoded in provided configs as well as in variables for all the rest of the code. With Jinja2 and env variables support we could just set the variables as env variables and then reference them in the configs. We could even randomize thise entities names to make the tests run on new entities each time and not run into problems with GitLab having a delay with actually deleting resources, to which you run into when you have static resource names.

I am starting to love this idea! 😍

[weakcamel]

Yes, an ability to grab values from environment or to pass arbitrary Jinja variables on via CLI to gitlabform (e. g. -e foo=bar -e baz=qux) and so on would be great! The option with environment vars would probably be more useful if someone wanted to make the logs public (would not expose the secrets in command-line parameters).

The change in logging also sounds like a reasonable, simple way to hide the secrets.

[gdubicki]

Using Jinja2 templating everywhere would also be a nice workaround for YAML limitation: it can have anchor/alias for a mapping/object, but not for a sequence.

We would just define the sequence as a variable and then put it into places where we want it using Jinga2 tags.

[codrcodz]

We wrapped a few tools together into a CLI and threw it into a docker container:

• https://gitlab.com/dreamer-labs/repoman/repoman

The tools include:

• gitlabform for managing the configuration of existing repositories
• python-gitlab for creating and deleting repositories and groups
• ansible-vault for managing secrets

Feel free to make your own wrapper based on ours, or contribute. MRs welcome!