Syncing vault secrets to k8s secrets.
This repository contains the code and deployment manifests for a Kubernetes controller, which will automate the process of syncing Vault secrets into Kubernetes secrets.
- How does the operator work
- Getting started
- Configuration
- VaultSecret manifest
- FAQ
- Reader tool
- Debugging
- Contributing
The operator watches for changes on VaultSecret
objects. Once a new VaultSecret
is created or an existing is modified, the operator will receive a notification in its reconcile loop
. Once inside the loop, it will:
- check validity of
VaultSecret
- populate a list of Vault paths (expand recursive paths to a list of absolute paths)
- read values from all paths and look for overrides (in case of override, it will not sync anything)
- store combined values into a Kubernetes Secret in either JSON, ENV or YAML format
- schedule another iteration of the loop after
reconcilePeriod
You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster.
Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info
shows).
- Install Instances of Custom Resources:
kubectl apply -f config/samples/
- Build and push your image to the location specified by
IMG
:
make docker-build docker-push IMG=<some-registry>/k8s-vault-operator:tag
- Deploy the controller to the cluster with the image specified by
IMG
:
make deploy IMG=<some-registry>/k8s-vault-operator:tag
To delete the CRDs from the cluster:
make uninstall
UnDeploy the controller to the cluster:
make undeploy
There are several configuration options available to customize a deployment of an operator. They are set in 3 different places:
- default values are hardcoded into the operator itself
- remote Kustomize base of the operator contains a
ConfigMap
with explicit default values that can be generally used on all clusters - cluster-specific Kustomize overrides in
ConfigMap
in infra cluster repository
Available configuration options:
OPERATOR_NAME
: a unique name for the operator - when running inside a cluster, it also serves as the name of the lockLOG_LEVEL
(defaultINFO
): specifies the ammount of logging output (values can beINFO
,DEBUG
)VAULT_ADDR
(defaulthttp://127.0.0.1:8200
): Vault address.DEFAULT_SA_AUTH_PATH
(no default): this value has to be assigned per cluster, and it specifies the default Vault path used for SA/JWT authentication- by default, it should follow this convention:
auth/k8s/<cluster>/login
- by default, it should follow this convention:
DEFAULT_RECONCILE_PERIOD
(default10m
): default reconcile period (i.e. how often will Vault secrets be synced)
Operator deployment injects environment variables from two Kubernetes Secrets:
system/vault-operator-env
: user-specified configuration (described above)
Syncing is performed with a VaultSecret
CRD (Custom Resource Definition).
Here you can see a minimum required example (with default values at bottom):
apiVersion: k8s.kiwi.com/v1
kind: VaultSecret
metadata:
name: test
namespace: my-namespace
spec:
paths:
- path: kw/secret/infra/platform/my-cluster/my-namespace/recursive/path/*
- path: kw/secret/infra/platform/my-cluster/my-namespace/my/sub/path/my-secret
prefix: my_prefix
# Those are defaults - you don't need to specify them!
separator: "_"
targetFormat: env
reconcilePeriod: 10m
targetSecretName: test # same as VaultSecret name
addr: http://127.0.0.1:8200
auth:
serviceAccountRef:
name: vault-operator-sync
authPath: auth/kw/infra/platform/my-cluster/my-namespace/jerry/login
role: my-namespace # same as namespace
Quick summary:
spec.addr
: Vault address used for authentication and fetching of Vault secretsspec.separator
: this string is used as a separator/delimiter when outputting in env formatspec.paths.[].path
: a path to a Vault secret or a partial/recursive path to a Vault sub-pathspec.paths.[].prefix
: a prefix that will be applied to all valuesspec.targetSecretName
: name of Kubernetes Secret where secrets will be synced intospec.targetFormat
: output format of synced secretsspec.reconcilePeriod
: amount of time between syncsspec.auth.serviceAccountRef.name
: name of Service Accountspec.auth.serviceAccountRef.authPath
: Vault path used for Service Account authenticationspec.auth.serviceAccountRef.role
: Vault role used for Service Account authentication
All details about the spec
are described in the following sections
Authentication setup is not handled by operator and needs to be done manually. Great docs on Hashiscorp website.
Quick setup for minikube and running operator on host machine:
vault auth enable kubernetes
kubectl apply -f config/samples/sa.yaml
export VAULT_SA_NAME=vault-operator-sync-token-secret
export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data.token}" | base64 --decode; echo)
export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath="{.data['ca\.crt']}" | base64 --decode; echo)
export K8S_HOST="localhost:8080"
kubectl proxy --port=8080
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="http://$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
vault policy write mypolicy config/samples/policy.hcl
vault write auth/kubernetes/role/vault-operator \
bound_service_account_names=vault-operator-sync \
bound_service_account_namespaces=default \
policies=default,mypolicy \
ttl=24h
Now you can create secret in Vault, e.g. secret/mysecret
and kubectl apply -f ./config/samples/v1_vaultsecret.yaml
.
Afterwards you should see K8s secret.
Full example:
auth:
serviceAccountRef:
name: operator-test
authPath: auth/kw/infra/platform/my-cluster/my-namespace/jerry/login
role: operator-test
Minimum required:
auth:
serviceAccountRef:
name: operator-test
Of the three values in auth.serviceAccountRef
, only name
is required and has to be set to the name of the Service Account used for Vault authentication. In most cases this will be the same as the name of Kubernetes Namespace. While this value could be infered, it is intentionally non-optional to force developers into thinking about Vault authentication.
authPath
will default toDEFAULT_SA_AUTH_PATH
(operator config)role
will default to the name of the Kubernetes Namespace
The VaultSecret
might have multiple paths defined. The values of paths are merged into one
kubernetes secret. If some path doesn't exist, it's skipped and vault operator create error log about this.
Also keys they're not matching naming convention (only A-Z
, a-z
, 0-9
, and -_
for key name) are skipped
excluded from kubernetes secrets. If we have paths with same key name, the value of the secret
is overridden, but we don't ensure in which order.
There are two different kinds of paths you can specify:
- paths to Vault secrets
- as absolute paths
- as recursive paths
Several topics listed below will show examples of how the operator will combine secrets into their final output. They will reference the following simplified Vault structure:
secrets/frontend/config
containsAPI_KEY=1
andBACKEND_ENDPOINT=2
secrets/backend/db/config
containsUSERNAME=3
andPASSWORD=4
secrets/backend/app/config
containsWORKERS=5
paths:
- path: secrets/frontend/config
The most basic case, where you define an absolute path to a Vault secret.
This has to be a full path, the same you would use if you used the vault
CLI tool:
vault kv get secrets/frontend/config
paths:
- path: secrets/backend/*
Operator will recursively find all sub-paths and all secrets on those sub-paths. As with the absolute paths, the part before /*
has to be a full path.
*
can only be placed at the very end of a path - secrets/*/config
is invalid.
Searching for secrets/backend/*
will output (assuming spec.separator
is _
and spec.targetFormat
is env
):
db_config_USERNAME=3
db_config_PASSWORD=4
app_config_WORKERS=5
Note that sub-paths and names of secrets are part of the output.
spec.paths.path.prefix
gives you the ability to customize how different paths combine with each other in their final output. Prefixes can have /
separators and should not be confused with spec.separator
, which are used only when outputting. Where you put /
separators in prefixes can have dramatic differences, especially in json
output.
To illustrate how prefixes can be leveraged, we'll go over the same example using multiple different prefixes.
Base example, without prefixes:
paths:
- path: secrets/frontend/config
- path: secrets/backend/*
API_KEY=1
BACKEND_ENDPOINT=2
db_config_USERNAME=3
db_config_PASSWORD=4
app_config_WORKERS=5
{
"API_KEY": 1,
"BACKEND_ENDPOINT": 2,
"db": {
"config": {
"USERNAME": 3,
"PASSWORD": 4
}
},
"app": {
"config": {
"WORKERS": 5
}
}
}
Prefixing with db_config
:
paths:
- path: secrets/frontend/config
prefix: db_config
- path: secrets/backend/*
Pay attention to the missing _
between db_config
and API_KEY
.
db_configAPI_KEY=1
db_configBACKEND_ENDPOINT=2
db_config_USERNAME=3
db_config_PASSWORD=4
app_config_WORKERS=5
{
"db_configAPI_KEY": 1,
"db_configBACKEND_ENDPOINT": 2,
"db": {
"config": {
"USERNAME": 3,
"PASSWORD": 4
}
},
"app": {
"config": {
"WORKERS": 5
}
}
}
Prefixing with db_config/
:
paths:
- path: secrets/frontend/config
prefix: db_config/
- path: secrets/backend/*
db_config_API_KEY=1
db_config_BACKEND_ENDPOINT=2
db_config_USERNAME=3
db_config_PASSWORD=4
app_config_WORKERS=5
{
"db_config": {
"API_KEY": 1,
"BACKEND_ENDPOINT": 2
},
"db": {
"config": {
"USERNAME": 3,
"PASSWORD": 4
}
},
"app": {
"config": {
"WORKERS": 5
}
}
}
Prefixing with db/config
:
paths:
- path: secrets/frontend/config
prefix: db/config
- path: secrets/backend/*
Pay attention to the missing _
between db_config
and API_KEY
.
db_configAPI_KEY=1
db_configBACKEND_ENDPOINT=2
db_config_USERNAME=3
db_config_PASSWORD=4
app_config_WORKERS=5
{
"db": {
"configAPI_KEY": 1,
"configBACKEND_ENDPOINT": 2,
"config": {
"USERNAME": 3,
"PASSWORD": 4
}
},
"app": {
"config": {
"WORKERS": 5
}
}
}
Prefixing with db/config/
:
paths:
- path: secrets/frontend/config
prefix: db/config/
- path: secrets/backend/*
db_config_API_KEY=1
db_config_BACKEND_ENDPOINT=2†
db_config_USERNAME=3
db_config_PASSWORD=4
app_config_WORKERS=5
{
"db": {
"config": {
"API_KEY": 1,
"BACKEND_ENDPOINT": 2,
"USERNAME": 3,
"PASSWORD": 4
}
},
"app": {
"config": {
"WORKERS": 5
}
}
}
spec.targetSecretName
defines the name of Kubernetes Secret, where Vault secrets will be synced into. It will be created in the same Kubernetes Namespace where VaultSecret
is.
There are several different output formats and spec.targetFormat
defines which one will be used.
You can use:
env
json
yaml
spec.reconcilePeriod
defines how often the operator will attempt to sync secrets. Default value is set to 10 minutes, which should be good for most cases.
Note: a fast reconcile period, along with a complex path structure, can cause a lot of requests to Vault. Keep this in mind when specifying this value.
Include the manifest in kustomization.yaml
in your overlay
as a resource
:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
...
resources:
- name-of-your-vaultsync-manifest.yaml
There are several checks made, to prevent syncing incomplete or invalid secrets.
- key in a Vault secret contains invalid characters (example:
a/b
is valid key in Vault, but cannot be used in Kubernetes) - Vault path does not exist
- VaultSecret manifest is invalid (use Reader tool to help with debugging)
- VaultSecret was re-applied too quickly (less than 10 seconds since last reconcile)
- wrongly configured Vault authentication (example: wrong
spec.auth.serviceAccountRef.name
) - overrides have been detected (two keys from different Vault paths override each other, use Reader tool to help with debugging)
We recommend to check vault operator logs and events with command:
kubectl get events -o json -n {desired_namespace} --field-selector involvedObject.kind=VaultSecret | jq '.items[].message'
It updates the status of VaultSecret
resource with LastUpdate
field and adds an event
to VaultSecret
. You can see both status and events with kubectl describe vaultsecret test
.
It is also part of operator logs, there will be message like:
Secret exists, data not equal, updating: gds-queue-handler Secret.Name: gds-queue-handler-secrets
The reader
tool has been developed to help create and debug VaultSecret
manifests. It will parse a manifest, connect to Vault and output the results to stdout
. Use this tool to configure your paths
and prefixes
before deploying VaultSecret
manifests.
reader
does not authenticate using Service Account, instead it uses Token authentication.
To use reader
, you must provide VAULT_ADDR
and VAULT_TOKEN
environment variables.
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=mysecrettoken
Note: your token will require permissions to access the paths specified in VaultSecret
you are testing.
-path
: path toVaultSecret
manifest you are testing-state
: path to state file, file does not have to exist on first run-o
: output format, defaults toenv
To view the operator logs, connect to your cluster, run kubectl get pod -n system
and look for:
...
vault-operator-v1-<RANDOM_ID> 1/1 Running 0 3h4m
...
Next, run kubectl logs -f vault-operator-v1-<RANDOM_ID> -n system
.
You will see output of every Secret
and VaultSecret
which passes through the operator. If the operator performs an update, you will see output like this:
{"level":"info","ts":"2023-01-16T14:02:00+01:00","msg":"Reconciling VaultSecret","controller":"vaultsecret","controllerGroup":"k8s.kiwi.com","controllerKind":"VaultSecret","VaultSecret":{"name":"vaultsecret-sample","namespace":"default"},"namespace":"default","name":"vaultsecret-sample","reconcileID":"f0780ec1-d164-4699-96c9-4d9e3e7befd7"}
{"level":"info","ts":"2023-01-16T14:02:00+01:00","msg":"Secret exists, data not equal, updating: default Secret.name: secrets-from-vault","controller":"vaultsecret","controllerGroup":"k8s.kiwi.com","controllerKind":"VaultSecret","VaultSecret":{"name":"vaultsecret-sample","namespace":"default"},"namespace":"default","name":"vaultsecret-sample","reconcileID":"f0780ec1-d164-4699-96c9-4d9e3e7befd7"}
{"level":"info","ts":"2023-01-16T14:02:00+01:00","msg":"Finished reconciling VaultSecret","controller":"vaultsecret","controllerGroup":"k8s.kiwi.com","controllerKind":"VaultSecret","VaultSecret":{"name":"vaultsecret-sample","namespace":"default"},"namespace":"default","name":"vaultsecret-sample","reconcileID":"f0780ec1-d164-4699-96c9-4d9e3e7befd7"}
Operator will save errors and certain key checkpoints as events
to VaultSecret
it currently syncs. To view them, run kubectl describe vaultsecret myvaultsecret -n mynamespace
. Multiple events of the same type will be merged together in this output.
Old events will be removed after some time, customizable per cluster and on GKE this is set to 1 hour.
// TODO(user): Add detailed information on how you would like others to contribute to this project
- Link the issue
- Squash commits
- Lint your code
- Include relevant test updates/additions
This project aims to follow the Kubernetes Operator pattern
It uses Controllers which provides a reconcile function responsible for synchronizing resources untile the desired state is reached on the cluster
- Install the CRDs into the cluster:
make install
- Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):
make run
NOTE: You can also run this in one step by running: make install run
If you are editing the API definitions, generate the manifests such as CRs or CRDs using:
make manifests
NOTE: Run make --help
for more information on all potential make
targets
More information can be found via the Kubebuilder Documentation
- Choose your kube context
- Run
make install
- this will install CRDs into your K8s cluster - Spin up vault server - eg.
docker-compose up vault
- Run
make run
- this will spin up operator on your machine- alternative
make deploy
- deploy controller to your cluster
- alternative
- Sample vault secret manifest is in
config/samples/v1_vaultsecret.yaml
kubectl apply -f config/samples/v1_vaultsecret.yaml
Tests are written in form of cases. Each case consists of vault_secret.yaml
and expected results:
expected.env
expected.json
expected.error
In vault_secret.yaml
you need to specify only paths
inside spec
. name
, namespace
, token
is overridden by test itself.
For each target format (env, json) VaultSecret manifest is created.
Tests are using test-env for k8s cluster and Vault server itself. So in order to run tests you need run Vault server:
- Using
docker-compose up vault
- Local vault server
export VAULT_DEV_ROOT_TOKEN_ID=testtoken
vault server -dev -dev-listen-address=0.0.0.0:8200
Afterwards just run make test
.
Read the full version Code of Conduct.
Copyright 2023.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.