Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Introduction to the Secretless Broker on Kubernetes

Description: Secretless Broker

Secrets are used to provide privileged access to protected resources. The Secretless Broker pushes the trust boundary of secrets away from application code into a privileged process that's designed with security best practices in mind. The Secretless Broker provides a local interface for application code to transparently consume protected resources.

Usage: Secretless Broker as Sidecar

The Secretless Broker operates as a sidecar container within a kubernetes application pod. This means there is shared storage/network between the application container and the Secretless Broker. It is this which allows the Secretless Broker to provide a local interface.

In this tutorial, we will walk through creating an application that communicates with a password-protected PostgreSQL database via the Secretless Broker. The application does not need to know anything about the credentials required to connect to the database; the admin super-user who provisions and configures the database will also configure the Secretless Broker to be able to communicate with it. The developer writing the application only needs to know the socket or address that the Secretless Broker is listening on to proxy the connection to the PostgreSQL backend.

To accomplish this, we are going to do the following:

As the admin super-user:

  1. Provision protected resources
  2. Configure protected resources for usage by application and add credentials to a secret store
  3. Configure the Secretless Broker to broker the connection using credentials from the secret store
  4. Create application identity and grant entitlements to provide access to credentials from the secret store

As the application developer:

  1. Configure the application to connect to protected resource through the interface exposed by the Secretless Broker
  2. Deploy and run the Secretless Broker adjacent to the application


The tutorial uses an existing pet store demo application that exposes the following routes:

  • GET /pets to list all the pets in inventory
  • POST /pet to add a pet
    • Requires Content-Type: application/json header and body that includes name data

There are additional routes that are also available, but these are the two that we will focus on for this tutorial.

Pet data is stored in a PostgreSQL database, and the application may be configured to connect to the database by setting the DB_URL, DB_USERNAME, and DB_PASSWORD environment variables in the application's environment (following 12-factor principles).

We are going to deploy the application with the Secretless Broker to Kubernetes, configure the Secretless Broker to be able to retrieve the credentials from a secrets store, and configure the application with the DB_URL environment variable pointing to the Secretless Broker and no values set for the DB_USERNAME or DB_PASSWORD environment variables.


To run through this tutorial, you will need:

  • A running Kubernetes cluster (you can use Minikube to run a cluster locally)
  • kubectl configured to point to the cluster
  • Docker CLI

Our Kubernetes deployment manifests assume that you are using Minikube, so that for example ./etc/pg.yml and ./etc/quick-start.yml use NodePort to expose the services; you may prefer to use a LoadBalancer for a GKE cluster.

Suggested modifications for advanced demos

Once you have run through this tutorial, you may enjoy trying it with some modifications to make it more pertinent to you. Here are some suggestions for things to try:

  • We've provided a sample application for you to try with the Secretless Broker - but if you're interested in exploring further, you can try out replacing it with your own app. To do this, you'll want to:

    • Modify quick-start.yml:35 to use your own application image
    • Update to appropriately configure the PostgreSQL database for your own application
  • You can use your own PostgreSQL database rather than using the database we deploy in this demo; for information on how to do this, please see "Option 2" of the provision database step.

Steps for the admin-level user

The following steps would be taken by an admin-level user, who has the ability to create and configure a database and to add secret values to a secret store.

These steps make use of the file, which stores the database connection info for the PostgreSQL backend.

1. Provision database

  • Provision protected resources

    [Option 1] PostgreSQL inside k8s

    Run the following script to deploy a PostgreSQL instance using a StatefulSet in the quick-start-backend-ns namespace:

    >>--- Clean up quick-start-backend-ns namespace
    Error from server (NotFound): namespaces "quick-start-backend-ns" not found
    namespace "quick-start-backend-ns" created
    secret "quick-start-backend-certs" created
    >>--- Create database
    statefulset "pg" created
    service "quick-start-backend" created
    Waiting for quick-start-backend to be ready

    Note that the PostgreSQL instance is deployed configured for SSL. We provide simple test certificates in this demo that we upload to the PostgreSQL container using Kubernetes Secrets. In practice, you will have your own certificates; for guidelines on creating the certificates for your PostgreSQL instance, check out the PostgreSQL documentation.

    [Option 2] Remote PostgreSQL server

    • Ensure your Kubernetes cluster is able to access your remote DB.

    • Ensure your remote instance supports SSL

    • Ensure the remote instance has a database called quick_start_db

    • Update the DB_ env vars in ./ For example (with Amazon RDS):

2. Configure database and add credentials to secret store

In this step, we will:

  • Configure the protected resources for usage by application (i.e. create DB user, add tables, etc.)
  • Add the application's access credentials for the database to a secret store


>>--- Set up database
>>--- Clean up quick-start-application-ns namespace
namespace/quick-start-application-ns created
secret/quick-start-backend-credentials created
serviceaccount/quick-start-application created created created

3. Configure the Secretless Broker to broker the connection to the target service

In the last step, we added the database credentials to our secret store - so to configure the Secretless Broker to be able to retrieve these credentials and proxy the connection to the actual PostgreSQL database, we have written a secretless.yml file that defines a PostgreSQL listener on port 5432 that uses the Kubernetes Secrets Provider to retrieve the credential values that we stored when we ran the last script:

  - name: pets-pg-listener
    protocol: pg
    address: localhost:5432

  - name: pets-pg-handler
    listener: pets-pg-listener
      - name: address
        provider: kubernetes
        id: quick-start-backend-credentials#address
      - name: username
        provider: kubernetes
        id: quick-start-backend-credentials#username
      - name: password
        provider: kubernetes
        id: quick-start-backend-credentials#password

Note: we don't specify an sslmode in the Secretless Broker config, so it will use the default require value.

Steps for the non-privileged user (i.e. developer)

Note: None of these steps require the information in - the person deploying the application needs to know nothing about the secret values required to connect to the PostgreSQL database!!

1. Configure application to access the database at localhost:5432

In the application manifest, we set the DB_URL environment variable to localhost:5432, so that when the application is deployed it will open the connection to the PostgreSQL backend via the Secretless Broker.

2. Deploy application

To deploy the application with the Secretless Broker, run:

>>--- Create and store Secretless configuration
configmap/quick-start-application-secretless-config created
>>--- Start application
deployment.apps/quick-start-application created
service/quick-start-application created
Waiting for quick-start-application to be ready

Try it out!

That's it! You've configured your application to connect to PostgreSQL via the Secretless Broker, and we can try it out to validate that it's working as expected.

Use the pet store app

POST /pet to add a pet - the request must include name in the JSON body


curl \
  -i \
  -d '{"name": "Mr. Snuggles"}' \
  -H "Content-Type: application/json" \
HTTP/1.1 201
Content-Length: 0
Date: Thu, 23 Aug 2018 12:57:45 GMT

GET /pets to retrieve notes


curl -i ${APPLICATION_URL}/pets
HTTP/1.1 200
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Thu, 23 Aug 2018 12:58:16 GMT

[{"id":1,"name":"Mr. Snuggles"}]

For convenience, we've provided the script for you to test adding a pet and retrieving the list of pets.

Rotate application database credentials

In addition to the demo you've seen so far, you can also rotate the DB credentials and watch the app continue to perform as expected.

The rotator script:

  • Rotates the credentials in the database
  • Updates the password in the secrets store
  • Prunes previously open connections

To see graceful rotation in action, poll the endpoint to retrieve the list of pets (GET /pets) in a separate terminal before rotating:


while true
    echo "Retrieving pets"
    curl -i ${APPLICATION_URL}/pets
    echo ""
    echo ""
    echo "..."
    echo ""
    sleep 3
Retrieving pets
HTTP/1.1 200
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Thu, 23 Aug 2018 12:58:43 GMT

[{"id":1,"name":"Mr. Snuggles"}]


To rotate the database password (note: you are acting as an admin user), run the following with your own value for [new password value]:

./ [new password value]
secret/quick-start-backend-credentials patched
(30 rows)

Observe that requests to the application API are not affected by the password rotation - we continue to be able to query the application as usual, without interruption!


If you enjoyed this Secretless Broker tutorial, please try to make it your own by trying out some of the suggested modifications. Please also let us know what you think of it! You can submit Github issues for features you would like to see, or send a message to our mailing list with comments and/or questions.