Vault Secrets Buildkite Plugins

This plugin was originally based on the the AWS S3 Secrets Buildkite Plugin

All secrets are base64 encoded in Vault It currently runs on an AWS based Buildkite stack, but it should work on any agent.

Expose secrets to your build steps. Secrets are stored encrypted-at-rest in HashiCorp Vault.

Different types of secrets are supported and exposed to your builds in appropriate ways:

• ssh-agent for SSH Private Keys
• Environment Variables for strings
• git-credential via git's credential.helper

Example

The following pipeline downloads a private key from https://my-vault-server/data/buildkite/{pipeline}/ssh_private_key and set of environment variables from https://my-vault-server/data/buildkite/{pipeline}/environment.

The private key is exposed to both the checkout and the command as an ssh-agent instance. The secrets in the env file are exposed as environment variables.

steps:
  - command: ./run_build.sh
    plugins:
      - mikeknox/vault-secrets#v0.1.:
          server: my-vault-server

Uploading Secrets

Secrets are uploading using the Vault CLI, as a base64 encoded blob in a field called value.

echo -n $(cat private_ssh_key | base64) | vault write  data/buildkite/test-pipeline/private_ssh_key \
  value=-

examples/ has 2 sample helper script for adding environment variables or ssh keys to Vault for a pipeline.

Environment secrets

Environment variable secrets are handled differently in this Vault plugin to the S3 plugin.

Each environment variable is treated as an individually secret under the env or environment nodes for a project. eg. project foo/env/var1 project foo/env/var2 etc

Policies

• Create policies to manage who can read and update pipeline secrets

The plugin needs at least read and list capabilities for the data. A sample read policy, this could be used by agents.

path "data/buildkite/*" {
    capabilities = ["read", "list"]
}

A sample update policy for build engineers or developers. This would allow creation of secrets for pipelines, but not as defaults.

# Allow update of secrets
path "data/buildkite/*" {
    capabilities = ["create", "update", "delete", "list"]
}
path "data/buildkite/env" {
    capabilities = ["deny"]
}
path "data/buildkite/environment" {
    capabilities = ["deny"]
}
path "data/buildkite/git-credentials" {
    capabilities = ["deny"]
}
path "data/buildkite/private_ssh_key" {
    capabilities = ["deny"]
}

SSH Keys

This example uploads an ssh key and an environment file to the base of the Vault secret path, which means it matches all pipelines that use it. You use per-pipeline overrides by adding a path prefix of /my-pipeline/.

# generate a deploy key for your project
ssh-keygen -t rsa -b 4096 -f id_rsa_buildkite
pbcopy < id_rsa_buildkite.pub # paste this into your github deploy key

export my_pipeline=my-buildkite-secrets
echo -n $(cat id_rsa_buildkite | base64) | vault write data/buildkite/my_pipeline/private_ssh_key \
    value=-

Git credentials

For git over https, you can use a git-credentials file with credential urls in the format of:

https://user:password@host/path/to/repo

vault write data/buildkite/my_pipeline/git-credentials value=- <<< $(echo "https://user:password@host/path/to/repo" | base64)

These are then exposed via a gitcredential helper which will download the credentials as needed.

Environment variables

Key values pairs can also be uploaded.

vault write data/buildkite/my_pipeline/environment value=- <<< $(echo "MY_SECRET=blah" | base64)

vault write data/buildkite/my_pipeline/env_key value=- <<< $(echo "my secret"| base64)

Can be loaded using:

steps:
  - command: ./run_build.sh
    plugins:
      - vault-secrets#v0.1.0:
          server: my-vault-server
          secrets:
          - key

Options

path

defaults to: data/buildkite This is expected to be a kv store

Alternative Base Path to use for Vault secrets

Testing

To run locally:

BUILDKITE_PLUGIN_VAULT_SECRETS_DUMP_ENV=true BUILDKITE_PLUGIN_VAULT_SECRETS_ADDR=http://0.0.0.0:8200 BUILDKITE_PIPELINE_SLUG=my_pipeline hooks/environment

To test with BATS:

docker-compose -f docker-compose.yml run --rm tests

Integration test:

.buildkite/steps/test_envvar.sh

When writing test plans, note that secrets are processed in the order they appear in the list returned from the Vault.

TODO

• Document use of TESTER_VAULT_VERSION version to set Vault version on tester service

• Add SVC_VAULT_VERSION to specify version of Vault service

• Document use of Makefile

• Merge compose files together

License

MIT (see LICENSE)