This repo contains a library that configures your secrets provider for something called "Managed Secrets".
Managed Secrets are an interface on your secrets provider. HashiCorp Vault is the reference implementation, but there's nothing stopping us from adding modules for AWS's Secrets Manager, SSM Parameter Store, or any other storage backend.
The idea behind Managed Secrets is that your secrets are managed
. Users do not need to know where the secrets are stored within the backend, how it's configured, or even what it is. They don't need to know anything about the backend at all. What's more, the backend could be swapped out at any time and nobody should notice or care.
Secrets management is an important job, but it sucks. Why does it suck?
First off, anyone who's actually run/managed a secrets system knows that making sure your app has the secrets it needs is just the tip of the iceberg. Yeah, that's all the developers care about, and that's all most people think of. Just about any solution can provide for that use case.
For the unsung heroes who maintain the system however, there's more. What? Here's a short list:
-
Audit (who has access to what?)
-
Logging (Who accessed what when?)
-
'Rotation' (i.e. changing secrets)
-
Granting of access
-
Revocation of access
-
Generating hard to guess secrets
The problem with the above tasks is, they're necessary, but they're time consuming and inglorious, so most of the time, they don't get done.
This is the point of Managed Secrets. Make the necessary painless, so you have a chance at actually doing it.
Secrets are defined as string values that must be kept from disclosure to unauthorized entities.
Secrets are key-value pairs that are owned by a Team. One secret 'key' has multiple values, one for each Environment (e.g., development, staging, or production, though there is no limit on the names or numbers of environments). The secrets
client can be configured to be environmentally aware based on CIDRs of client machines, and can automatically return the appropriate value for an Environment. You can specify different secrets for different environments, or just use the same secret name for all environments.
You gain access to Secrets via Roles. The Roles define which Principal can access which Secret in a given Environment. While Roles have no functional significance in terms of which Principals have access to which Secrets, and it is possible to put all such combinations inside a single Role, Roles function as an administrative aid to keep specified combinations of Principals, Secrets, and Environments logically grouped together for human interpretation.
The details of your client application will determine how you attach a Principal identity to that client. If using an IAM role as a Principal in the 'iam realm' (see below), for example, you could attach the IAM role to an EC2 instance to allow all applications on the instance to access your Secrets.
Each Secret has a 'generator' which can create and recreate the value of the Secret in each Environment (with the exception of static secret values, which are manually entered by an admin; see below). This is used to initially provision a Secret, and to rotate it as needed.
Ultimately, access to Secrets is controlled entirely by human code review for changes to this repo. As an example, developers on Team A can request access to Secrets owned by Team B simply by creating a Role in Team B's yaml file which provides access to that Secret to a principal controlled by Team A, such as an IAM role. There are no additional automated controls that prevent access from being granted. The only control is a human one: an admin will confirm with Team B that they intend to grant access to one of their secrets. This is a deliberate design feature. It is also something that could absolutely be used by evil conspirators on different teams. However well/poorly code review is handled will determine how 'safe' secrets access really is. Can everyone get together and agree that everyone gets access to everything? They sure can. Is that a good idea? In a production environment, probably not.
Based on the contents of a config file, these libs do the following:
-
Defines Secrets and their Generators for a Team.
-
Generate Secrets for a Team in each Environment, storing the Secrets in Vault.
-
Creates Vault Policies allowing Principals to access the above Secrets in each Environment.
-
Creates Roles per Team, generating Vault Auth endpoints allowing the
secrets
client or any other Vault savvy user to authenticate to Vault and get a token. -
Roles have 'Realms' which are computing environments. Each Realm configures a different flavor of Authentication backend. Choices are 'k8s', 'iam', and 'tls'.
-
Does not create the per team secrets engines in Vault. That has to be done manually by a Vault Admin. This is deliberate, and allows
keymaster
to run with limited permissions (creating new secrets engines would requirekeymaster
to run with root permissions). -
At present, CA engines are general purpose - not per Team.
This repo contains the libraries behind https://github.com/scribd/keymaster-cli. keymaster-cli
would be run against a set of config files as described above to configure Vault for Managed Secrets.
An example client for Managed Secrets is https://github.com/scribd/secrets. It leverages libraries in https://github.com/scribd/vault-authenticator to attempt parallel login methods to Vault, and then grab secrets from your Role.
Under Managed Secrets, your users take your version of secrets
, and either bake it into your containers, or use a dynamic binary system such as dbt to inject secrets
into containers at runtime.
With secrets
providing your access, and keymaster-cli
configuring Vault for you, you can make Secrets for your organization a self-service proposition while you do more interesting things.
Some Secrets cannot be generated, and must be manually placed into the storage backend. Third party API credentials are a good example of such 'static' secrets.
In the case of static secrets, the appropriate 'bucket' will be created in Vault when a configuration is merged, but any values will be an empty string. Note, however, that the Generator that creates values for other types of Secrets still operates on the storage bucket, and also creates a 'generator' value string for use in comparing buckets created at different times.
Although not strictly necessary, it is probably easier to use LDAP authentication for human users for entering static secrets manually. If you have another form of human user authentication configured for Vault, that works, too. The policy authorizations to secret paths for any authentication methods aside from IAM, Kubernetes, or TLS certificate are NOT configured by Managed Secrets. Thus, while you are able to create an (empty) static secret accessible by your applications via your yaml file, you will not be able to set the value of the secret unless your personal username is manually given write permissions to the specific Vault path. (This is the only scenario in which some knowledge of the Vault backend, namely, the exact path at which secrets provisioned by keymaster
are stored, is required, although all secrets belonging to a given theam are stored at the team_name/ path, making them easy to find.)
TLS certificates are handled much in the same way as any other secret, but they are multi-valued.
For a TLS Secret named 'foo.scribd.com', you should expect to find a 'foo.scribd.com.key', 'foo.scribd.com.crt', 'foo.scribd.com.serial', etc.
TLS certificate secrets are automatically renewed when they are near expiration. N.B.: At the time of this writing, this has not been implemented. The code to regenerate exists, but it's not wired up to anything.
The Roles defined in this repo have the power to consume TLS Secrets, but they cannot generate them. This is an important point. By separating generation from consumption, it severely limits the blast radius of a compromised application. The attacker can steal the credentials, but they cannot create new ones.
Create a directory for your Team. Put it under the path in this repo secrets
.
Name it whatever you like. Only yaml files in the directory secrets
in this repo are parsed and synced to Vault.
It's probably best to name the directory after your Team, but hey, you're the one that needs to keep track of it and communicate it to others. Knock yourself out.
Under that directory, create a yaml file (example below). The file name doesn't matter here, either. Again, the author recommends you name the file after your Team, but it's only a suggestion.
Each file defines a Team. The name
value at the top of this file will be the name of the Team's secret container. Even this name doesn't matter. Only one Team is allowed per file, but each directory can have multiple files. Note, however, that spreading a single Team's info across multiple files is not currently supported; each file in a directory must contain a different Team name at the top.
In your yaml file, define your Secrets, Roles, and Environments.
The Roles allow specified Principals to read specified Secrets. As noted above, Roles are merely an administrative aid to keep specified combinations of Principals and Secrets logically grouped together. To associate a single Role with different Principals and/or grant access to different secret values in different Environments, make multiple Realm blocks (specifying 'realms', 'principals', and 'secrets') for the same Role name (the 'name' of the Role can optionally be restated for readability). See the example below for the 'app1' Role.
Environments are merely the "buckets" that each secret is split into. If three different Environments are defined for a team, each secret will have three different buckets in which to place unique values for all the Secrets defined in the yaml, though they don't all need to be used. The names of the Environments can be anything, but the secrets
binary can be customized to treat recognize certain environments by CIDR. When a client has a source IP in a CIDR that corresponds to one of these environments, secrets
automatically recognizes the environment, and saves the calling process from needing to specify the -e
flag with secrets
.
Realms have 'types' ('k8s', 'iam', 'tls'), as well as 'principals'. The 'identifiers' list is only used by the 'k8s' type and is the name of the Kubernetes Cluster.
The possible 'principals' are:
- the ARNs of the IAM role (or IAM user) for the 'iam' type
- the FQDN of the host for the 'tls' type
- the namespace for the 'k8s' type (the
secrets
tool uses thedefault
service account in the specified namespace to authenticate)
Note: a Managed Secrets Role (defined by this README) is unrelated to an AWS IAM role (a technical AWS term). A single IAM role could be specified as a Principal in multiple Managed Secrets Roles. However, keeping the scope of both types of roles the same (i.e., logically mapping one Managed Secrets Role to one AWS IAM role) makes it easier to keep track of which IAM roles have access to which Secrets. Giving them each the same name helps, too.
Note: although LDAP authentication to Vault is possible, it isn't one of the authentication methods that is configured by Managed Secrets automation. A Vault admin must manually configure a specific Vault policy to allow LDAP authentication. How you set this up is dependent on how you assign users to LDAP groups.
Once you have your secrets defined, open a PR to master. Scribd configures its CI to check yaml files for syntax when a PR is opened.
Get someone to review your PR.
Keeping it all secure and sane is everyone's responsibility.
Make a Jira task, send a Slack message, etc.
Some manual setup must be completed by a Vault admin before a new Managed Secrets configuration can be provisioned. This was a deliberate choice (see "Capabilities" section, above).
A Vault admin must configure the storage for your secrets before you merge to master or CD will fail. See Step 7.
This example defines a Team
---
name: test-team1 # The name of the Team
secrets: # Definitions of the Secrets in the Team
- name: foo
generator:
type: alpha # A 10 digit alphanumeric secert
length: 10
- name: bar
generator:
type: hex # A 12 digit hexadecimal secret
length: 12
- name: baz
generator:
type: uuid # A UUID secret
- name: wip
generator:
type: chbs
words: 6 # A 6 word 'correct-horse-battery-staple' secret. 6 random commonly used words joined by hyphens.
- name: zoz
generator:
type: rsa
blocksize: 2048 # A RSA keypair expressed as a secret (Not currently supported)
- name: blort # I've clearly run out of standard throwaway names here.
generator:
type: static # Static secrets have to be placed manually. API keys are a good use case for Static Secrets.
- name: foo.scribd.com
generator:
type: tls # A TLS Certificate/ Private Key expressed as a secret.
cn: foo.scribd.com
ca: service # This cert is created off of the 'service' CA
sans:
- bar.scribd.com # Allowed alternate names for this cert
- baz.scribd.com
ip_sans: # IP SANS allow you to use TLS and target an IP directly
- 1.2.3.4
roles: # Your Secret Roles This is what you authenticate to in order to access the Secrets above.
- name: app1 # A role unimaginatively named 'app1'; NOT case sensitive
realms:
- type: k8s # legal types are 'k8s', 'tls', and 'iam'
identifiers:
- bravo # for k8s, this is the name of the cluster. Has no meaning for other realms.
principals:
- app1 # for k8s, this is the namespace that's allowed to access the secret
environment: production # each role maps to a single environment. Which one is this?
- type: tls # 'tls' specifies authentication by client certs (generally only applies to SL hosts)
principals:
- foo.scribd.com # for tls, this is the FQDN of a host that possesses a root CA-signed certificate
environment: development # when this host connects, it gets development secrets
- type: iam # only works if the client has an IAM identity, which usually means the client
principals: # is running inside AWS
- "arn:aws:iam::<team-stage-account>:role/foo"
- "arn:aws:iam::<team-stage-account>:user/foo"
- "arn:aws:iam::<team-stage-account>:role/foo-*" # wildcards allowed!
- "arn:aws:sts::<team-stage-account>:assumed-role/foo/*" # allows entities that can assume the role to authenticate
environment: staging # each principal auths to a role in a single environment.
secrets:
- name: foo # These Secrets are defined above. No 'team' in the config means 'team from this file'
- name: wip
- name: baz
team: test-team2 # This secret is owned by another Team.
- name: app1 # The "realms:" type and/or the principals can be modified in repeated blocks beneath role names
realms: # to give the same (or different) principals access to different versions of the same (or different)
- type: iam # secrets in different environments. This example is a "maximum" differential of
principals: # principal, environment, and secret names, but it is also possible to change just
- "arn:aws:iam::<team-prod-account>:role/foo" # one or two of these parameters
environment: production # Restating the "name" and "realms:" lines is not necessary, but increases readability
secrets:
- name: blah # The top two secrets are completely different than "foo" and "wip", above
- name: bloo
- name: wip # This is the same secret as above, but refers to the "production" version
# instead of the "staging" version
environments:
- production
- staging
- development
The machine running keymaster
requires broad Vault permissions to configure secrets. We don't list them here, but the Vault API operations conducted by keymaster
are readily apparent in the code. Additionally, keymaster
requires create and update access to each new secrets engine enabled (and all subpaths), the AWS authentication method path, and each new Kubernetes authentication method path created.
The libraries cannot enable Vault secrets engines. The root token is required for this operation. Relying on a human user to enable a secrets engine both removes the requirement for the libraries to be run as root, and forces a human to take an affirmative action to approve an onboarding of a new team.
Every time a new team is onboarded to Managed Secrets, an admin will need to manually run:
vault secrets enable -version=2 -path=<team name> -description="<team name> Managed Secrets" kv
Keymaster does not remove deprecated secrets or Managed Secrets roles. Although it would likely be trivial to fork keymaster
and add functionality to automatically delete secrets and roles based solely on their removal from a yaml file, we do not recommend doing this. Secret values and secret access authorization configurations are some of the most sensitive data in any environment. Retaining deletion authorization for a human user reduces the risk of loss of potentially irreplaceable information due to compromise of a CD system.
This isn’t necessary for the new or renamed role or secret to work, but over time, it will lead to a proliferation of unused roles and secret paths inside the storage backend, which will make auditing (and troubleshooting!) more difficult.
If a Managed Secrets role or secret name has changed or removed, you should manually remove the access authorization and secret. Using the reference Vault backend, this requires deleting the Vault policy corresponding to the deprecated role and the environment(s) for which it authorized secret access, and/or the old secret paths:
$ vault policy delete <team-name>-<old-role-name>-<environment>
$ vault kv destroy <team-name>/<secret-name>
Before removing a secret path with kv destroy
, ensure you have moved the secret values to the new secret path. If the secret value will no longer be used, but you wish to retain the value and version history, use vault kv delete <team-name>/<secret-name>
Using the IAM and Kubernetes authentication methods requires some understanding of these systems that is beyond the scope of Managed Secrets. HashiCorp has voluminous documentation on reference architectures for these authentication methods. Some necessary, but possibly not sufficient, key points for you to implement Managed Secrets using Vault as a storage backend:
Initial points to avoid confusion:
- "Vault permissions" and "IAM permissions" are two different things, and we've tried to make clear which one we're talking about below
- Some necessary Vault permissions and all necessary IAM permissions are manually configured, i.e.,
keymaster
does not configure IAM permissions or any other AWS infrastructure
Morever, although it would likely be trivial to fork keymaster
and add functionality to automatically configure AWS as well as Vault based solely on the yaml file, we do not recommend doing this. As with secret and role deletion, above, separating the Vault configuration from the AWS configuration creates a barrier between catastrophic compromise of your secrets. It also lends itself to having a human admin in the loop reviewing secret setups of various developer teams.
The machine that runs Vault (and therefore Vault itself) must be able to call sts:GetCallerIdentity
in the AWS account in which it exists (this is configured by the current HashiCorp reference Terraform). Vault must also be able to assume an IAM role in the account that contains the IAM principal specified in the Managed Secrets yaml. This assumed role must be granted the ec2:DescribeInstances
,iam:GetInstanceProfile
,iam:GetUser
, and iam:GetRole
permissions. This assumed role, its permissions, and the permission for Vault to assume it must be configured in AWS, independently of Managed Secrets. Additionally, this role must be manually specified in Vault as an "STS role".
A reference architecture and workflow is:
- an EC2 machine running
keymaster
in the same (ideally, dedicated) AWS account in which Vault runs, which - authenticates to Vault using IAM authentication (manually configured outside of Managed Secrets) and
- grants
keymaster
the Vault permission to make the appropriate API call to Vault to configure the specified IAM principal to retrieve secrets from Vault - Vault uses its default IAM role (from reference Terraform) with an IAM permission policy (manually configured) to assume an IAM role (manually configured, we'll call it the
VaultLookup
IAM role) in the AWS account that contains the specified IAM principal; thisVaultLookup
IAM role must be manually specified in Vault as an "STS role" (see above) - Vault makes various AWS API calls (using the
VaultLookup
role it has assumed) to validate that the specified IAM principal exists
If you fail to specify the VaultLookup
role as an "STS role" in Vault, you'll receive the following error, which is unhelpful unless you are very familiar with AWS IAM terminology ("internal ID") and also understand the mechanics of how Vault integrates with IAM (using a "client"):
URL: PUT https://vault.foo.com/v1/auth/aws/role/specified-IAM-principal
Code: 400. Errors:
unable to resolve ARN "arn:aws:iam::<id of account containing IAM principal>:role/specified-IAM-principal" to internal ID: unable to fetch client for account ID "<id of account containing IAM principal>" -- default client is for account "<Vault account id>"
Each K8s authentication method is manually enabled in Vault at a unique path. This path must be added to the Vault permissions granted to keymaster
.
A CA-signed certificate must be hardcoded into your private fork of keymaster-cli
in order for TLS authentication methods to be configured.
Managed Secrets currently only configures the allowed_common_names
parameter in Vault based on the principals specified in a team’s yaml. It does not configure other restrictions, such as token_bound_cidrs
. Additional restrictions have to be manually configured after Managed Secrets has already configured the TLS role. For example, using the example yaml configuration above:
$ curl -XPOST -d @payload.json --header "X-Vault-Token: $(< .vault-token)" https://vault.foo.com/v1/auth/cert/certs/test-team1-app1
with payload.json
as:
{
"token_bound_cidrs": [
"10.1.2.3",
"10.2.3.4",
<additional IP addresses>
]
}
There are several other restrictions that can be configured manually. See Vault documentation. We'd like to extend keymaster
's ability to automatically configure these additional parameters. Make a PR!