Secure your apps by making them Secretless
Clone or download
izgeri Merge pull request #565 from cyberark/560-footer
Update footer link for Google Group
Latest commit 4f7143f Dec 7, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github Move K8s CRD tests into manual/ folder Nov 27, 2018
Contributing_OSS Add OSS Contributing Agreement Jul 6, 2018
bin Use newest goreleaser for building releases Dec 6, 2018
cmd Made configuration file reading use a configuration manager plugin Aug 30, 2018
demos Add convenience script for testing K8s demo Dec 4, 2018
design Add some notes to the TLS support plan Dec 3, 2018
docs Merge pull request #565 from cyberark/560-footer Dec 7, 2018
internal Update broken keychain provider test Nov 27, 2018
pkg Editing plugin godoc to improve readability Sep 28, 2018
resource-definitions Update the READMEs to centralize developer content Dec 4, 2018
test Update the keychain provider test Dec 4, 2018
.dockerignore Fix linting error handling Aug 29, 2018
.gitignore Add JetBrains IDE dir to .gitignore Nov 6, 2018
.gitlab-ci.yml Only save the latest build locally in GitLab Nov 26, 2018
.goreleaser.yml Updated goreleaser config to use the newest spec Dec 6, 2018
CHANGELOG.md Bumping to version 0.5.2 Nov 28, 2018
CONTRIBUTING.md Remove client-go lines from go.sum before build Dec 4, 2018
Dockerfile Add GH issue links to hacky removal of client-go Dec 6, 2018
Dockerfile.dev Add GH issue links to hacky removal of client-go Dec 6, 2018
Dockerfile.test Add GH issue links to hacky removal of client-go Dec 6, 2018
Jenkinsfile Remove unneeded array with cleaner approach Nov 7, 2018
LICENSE.md Added ASL 2.0 as the project license Jul 11, 2018
README.md Update the READMEs to centralize developer content Dec 4, 2018
STYLE.md Updating docs to refer to Secretless Broker Aug 8, 2018
VERSION Bumping to version 0.5.2 Nov 28, 2018
go.mod feat: secretless provider for aws secrets manager Nov 21, 2018
go.sum Update the keychain provider test Dec 4, 2018
secrets.yml Secretless has a website (#130) Jul 10, 2018

README.md

pipeline status


Status: Beta

The Secretless Broker is currently in beta.


Table of Contents

Secretless Broker

The Secretless Broker is a connection broker which relieves client applications of the need to directly handle secrets to target services such as databases, web services, SSH connections, or any other TCP-based service.

Secretless Broker Architecture

The Secretless Broker is designed to solve two problems. The first is loss or theft of credentials from applications and services, which can occur by:

  • Accidental credential leakage (e.g. credential checked into source control, etc)
  • An attack on a privileged user (e.g. phishing, developer machine compromises, etc)
  • A vulnerability in an application (e.g. remote code execution, environment variable dump, etc)

The second is downtime caused when applications or services do not respond to credential rotation and crash or get locked out of target services as a result.

When the client connects to a target service through the Secretless Broker:

  • The client is not part of the threat surface

    The client/app no longer has direct access to the password, and therefore cannot reveal it.

  • The client doesn’t have to know how to properly manage secrets

    Handling secrets safely involves some complexities, and when every application needs to know how to handle secrets, accidents happen. The Secretless Broker centralizes the client-side management of secrets into one code base, making it easier for developers to focus on delivering features.

  • The client doesn’t have to manage secret rotation

    The Secretless Broker is responsible for establishing connections to the backend, and can handle secrets rotation in a way that’s transparent to the client.

To provide the Secretless Broker access to a target service, a Handler implements the protocol of the service, replacing the authentication handshake. The client does not need to know or use a real password to the service. Instead, it proxies its connection to the service through a local connection to the Secretless Broker via a Listener. The Secretless Broker obtains credentials to the target service from a secrets vault (such as Conjur, a keychain service, text files, or other sources) via a Credential Provider. The credentials are used to establish a connection to the actual service, and the Secretless Broker then rapidly shuttles data back and forth between the client and the service.

The Secretless Broker is currently licensed under ASL 2.0

Currently supported services

  • MySQL (Socket and TCP)
  • PostgreSQL (Socket and TCP)
  • SSH
  • SSH-Agent
  • HTTP (Basic auth, Conjur, and AWS authorization strategies)

With many others in the planning stages!

If there is a specific target service that you would like to be included in this project, please open a GitHub issue with your request.

For specific guidelines about using a particular service, please see our instructions for using the Secretless Broker.

Quick Start

Running the quick start demo requires Docker.

To see the Secretless Broker in action, build the quick start image:

$ cd demos/quick-start/
$ ./bin/build
...
Successfully built cbf747e7f548
Successfully tagged secretless-broker-quickstart:latest

The quick start image runs the Secretless Broker binary and a few sample target services. The Secretless Broker is configured to retrieve the credentials required to access the services from the process environment. All services are configured to require authorization to access, but we don't know what those credentials are. We can try to access the services, but since we don't know the password our access attempts will fail. But when we try to connect via the Secretless Broker, we will be granted access.

Let's try this with the PostgreSQL server running in the quick start image. We know that the server has been configured with a quickstart database, so let's try to access it.

You can run the Secretless Broker quick start as a Docker container:

$ docker run \
  --rm \
  -p 5432:5432 \
  -p 5454:5454 \
  secretless-broker-quickstart:latest

In a new window, if we try to connect to PostgreSQL directly via port 5432 (guessing at the postgres username), our attempt will fail:

$ psql -h localhost -p 5432 -U postgres -d quickstart
Password for user postgres:
psql: FATAL:  password authentication failed for user "postgres"

But the Secretless Broker is listening on port 5454, and will add authentication credentials (both username and password) to our connection request and proxy our connection to the PostgreSQL server:

$ psql \
  -h localhost \
  -p 5454 \
  --set=sslmode=disable \
  -d quickstart
psql (10.3, server 9.6.9)
Type "help" for help.

quickstart=> \d
                List of relations
 Schema |      Name       |   Type   |   Owner    
--------+-----------------+----------+------------
 public | counties        | table    | secretless
 public | counties_id_seq | sequence | secretless
(2 rows)

quickstart=> select * from counties limit 1;
 id |   name    
----+-----------
  1 | Middlesex
(1 row)

quickstart=>

Success! Smile and grab a 🍪 because it was too easy!

You have just delegated responsibility for managing credentials to a secure process isolated from your app!

Run more Secretless Broker demos

If the PostgreSQL quick start demo piqued your interest, please check out our additional demos where you can try the Secretless Broker with SSH and HTTP Basic Auth.

For an even more in-depth demo, check out our Deploying to Kubernetes demo, which walks you through deploying a sample app to Kubernetes with the Secretless Broker.

Using Secretless

The Secretless Broker relies on YAML configuration files to specify which target services it can connect to and how it should retrieve the access credentials to authenticate with those services.

Each Secretless Broker configuration file is composed of two sections:

  • listeners: A list of protocol Listeners, each one on a Unix socket or TCP port.
  • handlers: A list of Handlers to process the requests received by each Listener. Handlers implement the protocol for the target services and are configured to obtain the backend connection credentials from one or more Providers.

Listeners

You can configure the following kinds of Secretless Broker Listeners:

  1. unix The Secretless Broker serves the backend protocol on a Unix domain socket.
  2. tcp The Secretless Broker serves the backend protocol on a TCP socket.

For example, PostgreSQL clients can connect to the PostgreSQL server either via Unix domain socket or over a TCP connection. If you are setting up the Secretless Broker to facilitate a connection to a PostgreSQL server, you can either configure it:

  • to listen on a Unix socket as usual (default: /var/run/postgresql/.s.PGSQL.5432)

    listeners:
    - name: pg_socket
      protocol: pg
      socket: /sock/.s.PGSQL.5432
    

    In this case, the client would be configured to connect to the database URL /sock.

  • to listen on a given port, which may be the PostgreSQL default 5432 or may be a different port to avoid conflicts with the actual PostgreSQL server

    listeners:
    - name: pg_tcp
      protocol: pg
      address: 0.0.0.0:5432
    

    In this case, the client would be configured to connect to the database URL localhost:5432

Note that in each case, the client is not required to specify the username and password to connect to the target service. It just needs to know where the Secretless Broker is listening, and it connects to the Secretless Broker directly via a local, unsecured connection.

In general, there are currently two strategies to redirect your client to connect to the target service via the Secretless Broker:

  1. Connection URL

    Connections to the backend service are established by a connection URL. For example, PostgreSQL supports connection URLs such as postgres://user@password:hostname:port/database. hostname:port can also be a path to a Unix socket, and it can be omitted to use the default PostgreSQL socket /var/run/postgresql/.s.PGSQL.5432.

  2. Proxy

    HTTP services support an environment variable or configuration setting http_proxy=<url> which will cause outbound traffic to route through the proxy URL on its way to the destination. The Secretless Broker can operate as an HTTP forward proxy, in which case it will place the proper authorization header on the outbound request. It can also optionally forward the connection using HTTPS. The client should always use plain http:// URLs, otherwise the Secretless Broker cannot read the network traffic because it will encrypted.

Regardless of the connection strategy, the operating system provides security between the client and the Secretless Broker. It's important to configure the OS properly so that unauthorized processes and clients can't connect to the Secretless Broker. With Unix domain sockets, operating system file permissions protect the socket. With TCP connections, the Secretless Broker should be listening only on localhost.

The Listener configuration governs the client to Secretless Broker connection. The connection from the Secretless Broker to the PostgreSQL server is defined in the Handler configuration, where the actual address and credential information for the connection to the PostgreSQL server is defined.

At this time, the Secretless-Broker-to-target-service connection always happens over TCP by default.

Handlers

When the Secretless Broker receives a new request on a defined Listener, it automatically passes the request on to the Handler defined in the Secretless Broker configuration for processing. Each Listener in the Secretless Broker configuration should therefore have a corresponding Handler.

The Handler configuration specifies the Listener that the Handler is handling connections for and any credentials that will be needed for that connection. Several credential sources are currently supported; see the Credential Providers section for more information.

In this example, I am setting up a Handler to process connection requests from the pg_socket Listener, and it has three credentials: address, username, and password. The address and username are literally specified in this case, and the password is taken from the environment of the running Secretless Broker process.

handlers:
  - name: pg_via_socket
    listener: pg_socket
    credentials:
      - name: address
        provider: literal
        id: pg:5432
      - name: username
        provider: literal
        id: myuser
      - name: password
        provider: env
        id: PG_PASSWORD

In production you would want your credential information to be pulled from a vault, and the Secretless Broker currently supports multiple vault Credential Providers.

When a Handler receives a new connection requests, it retrieves any required credentials using the specified Provider(s), injects the correct authentication credentials into the connection request, and opens up a connection to the target service. From there, the Handler simply transparently shuttles data between the client and service.

Please note: Handler API interface signatures are currently under heavy development due to needing to deal with non-overlapping types of communications protocols (as expressed by the interface definitions) so they will be likely to change in the near future.

Handler Credentials

The Secretless Broker comes with several built-in Handlers, and each accepts a different set of credentials for configuration. In this section we provide information on the credentials used by each Handler.

  • MySQL (accepts connections over Unix socket or TCP)
    • Credentials:
      • host
      • port
      • username
      • password
  • PostgreSQL (accepts connections over Unix socket or TCP)
    • Credentials:
      • address
      • username
      • password
  • SSH
    • Credentials:
      • address
      • privateKey
      • user
        • optional; defaults to root
      • hostKey
        • optional; accepts any host key if not included
  • SSH-Agent
    • Credentials:
      • rsa
      • ecdsa
      • comment
        • optional; free-form string
      • lifetime
        • optional; if not 0, the number of secs the agent will store the key for
      • confirm
        • optional; confirms with user before using if true
  • HTTP
    • Basic Auth
      • Credentials:
        • username
        • password
        • forceSSL (optional)
    • Conjur
      • Credentials:
        • accessToken
        • forceSSL (optional)
    • AWS
      • Credentials:
        • accessKeyID
        • secretAccessKey
        • accessToken

Credential Providers

Credential providers interact with a credential source to deliver secrets needed for authentication to the Secretless Broker Listeners and Handlers. The Secretless Broker comes built-in with several different Credential Providers, making it easy to use with your existing workflows regardless of your current secrets management toolset.

We currently support the following secrets providers/vaults:

Conjur Provider

Conjur (conjur) provider allows use of CyberArk Conjur for fetching secrets.

Example:

...
    credentials:
      - name: accessToken
        provider: conjur
        id: path/to/the/token
...

HashiCorp Vault Provider

Vault (vault) provider allows use of HashiCorp Vault for fetching secrets.

Example:

...
    credentials:
      - name: accessToken
        provider: vault
        id: path/to/the/token
...

File Provider

File (file) provider allows you to use a file available to the Secretless Broker process and/or container as sources of credentials.

Example:

...
    credentials:
      - name: rsa
        provider: file
        id: /path/to/file
...

Environment Variable Provider

Environment (env) provider allows use of environment variables as source of credentials.

Example:

...
    credentials:
      - name: accessToken
        provider: env
        id: ACCESS_TOKEN
...

Literal Value Provider

Literal (literal) provider allows use of hard-coded values as credential sources.

Note: This type of secrets inclusion is highly likely to be much less secure versus other available providers so please use care when choosing this as your secrets source.

Example:

...
    credentials:
      - name: accessToken
        provider: literal
        id: supersecretaccesstoken
...

Keychain Provider

Keychain (keychain) provider allows use of your OS-level keychain as the credentials provider.

Note: This provider currently only works on Mac OS at the time and only when building from source so it should be avoided unless you are a developer working on the source code. There are plans to integrate all major OS keychains into this provider in a future release.

Example:

...
    credentials:
      - name: rsa
        provider: keychain
        id: servicename#accountname
...

Community

Our primary channel for support is through our Secretless Broker mailing list. More info here: community support

Performance

Using the Secretless Broker reduces the transaction throughput by about 25% on Postgresql. Once the connection to the backend database is established, the Secretless Broker runs 2 goroutines - one reads from the client and writes to the server, the other reads from the server and writes to the client. It's as simple as this:

    go stream(self.Client, self.Backend)
    go stream(self.Backend, self.Client)

Here is some performance data created by running pgbench in a Dockerized environment with the client, the Secretless Broker and database running on a single machine (2017 MacBook Pro with 4-core Intel Core i7 @ 2.9GHz).

Directly to the database:

root@566b7c06abcf:/go/src/github.com/cyberark/secretless-broker# PGPASSWORD=test PGSSLMODE=disable pgbench -h pg -U test -T 10 -c 12 -j 12 postgres
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 12
number of threads: 12
duration: 10 s
number of transactions actually processed: 14434
latency average: 8.327 ms
tps = 1441.077559 (including connections establishing)
tps = 1443.230144 (excluding connections establishing)

Through the secretless-broker proxy:

root@566b7c06abcf:/go/src/github.com/cyberark/secretless-broker# PGSSLMODE=disable pgbench -h 172.18.0.9 -T 10 -c 12 -j 12 postgres
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 12
number of threads: 12
duration: 10 s
number of transactions actually processed: 10695
latency average: 11.237 ms
tps = 1067.933129 (including connections establishing)
tps = 1075.661082 (excluding connections establishing)

Here is a set of test results running directly against an RDS Postgresql:

root@2a33637a9cb5:/work# pgbench -h demo1.cb5uzm0ycqol.us-east-1.rds.amazonaws.com -p 5432 -U alice -T 10 -c 12 -j 12 postgres
Password:
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 12
number of threads: 12
duration: 10 s
number of transactions actually processed: 197
latency average: 657.775 ms
tps = 18.243307 (including connections establishing)
tps = 18.542609 (excluding connections establishing)

And to RDS through the Secretless Broker:

root@2a33637a9cb5:/work# pgbench -U alice -T 10 -c 12 -j 12 postgres
starting vacuum...end.
transaction type: TPC-B (sort of)
scaling factor: 1
query mode: simple
number of clients: 12
number of threads: 12
duration: 10 s
number of transactions actually processed: 153
latency average: 824.491 ms
tps = 14.554441 (including connections establishing)
tps = 15.822442 (excluding connections establishing)

14% fewer tps (excluding establishing connections) via the Secretless Broker.

Changing the -c (number of clients) and -j (number of threads) didn't have much effect on the relative throughput, though increasing these from 1 to 12 does approximately double the tps in both direct and proxied scenarios.

Development

We welcome contributions of all kinds to the Secretless Broker. For instructions on how to get started and descriptions of our development workflows, please see our contributing guide. This document includes guidelines for writing plugins to extend the functionality of Secretless Broker.

License

The Secretless Broker is licensed under Apache License 2.0 - see LICENSE.md for more details.